[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 22:21:46 dist-test-slave-ncp9 krb5kdc[9936](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 22:21:46 dist-test-slave-ncp9 krb5kdc[9936](info): set up 2 sockets
Aug 08 22:21:46 dist-test-slave-ncp9 krb5kdc[9936](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260808 22:21:47.468575 9923 mini_kdc.cc:121] Time spent starting KDC: real 1.122s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/krb5kdc/test-user.keytab.
Aug 08 22:21:47 dist-test-slave-ncp9 krb5kdc[9936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227707, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T22:21:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:21:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.176.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.176.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.176.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab.
Entry for principal kudu/127.9.176.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab.
I20260808 22:21:47.587688 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:37449
--webserver_interface=127.9.176.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab
--principal=kudu/127.9.176.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:36811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.176.254:37449
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:21:47.711596 9953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:21:47.711921 9953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:21:47.711990 9953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:21:47.716015 9953 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:21:47.716099 9953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:21:47.716130 9953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:21:47.716166 9953 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:21:47.716197 9953 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:21:47.722137 9953 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:36811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.176.254:37449
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab
--principal=kudu/127.9.176.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:37449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.176.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9953
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:21:47.723732 9953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:21:47.725109 9953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:21:47.738560 9963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:21:47.738637 9960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:21:47.738747 9961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:21:47.738606 9953 server_base.cc:1034] running on GCE node
I20260808 22:21:47.739537 9953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:21:47.741436 9953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:21:47.742641 9953 hybrid_clock.cc:648] HybridClock initialized: now 1786227707742625 us; error 61 us; skew 500 ppm
Aug 08 22:21:47 dist-test-slave-ncp9 krb5kdc[9936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227707, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:21:47.745720 9953 init.cc:377] Logged in from keytab as kudu/127.9.176.254@KRBTEST.COM (short username kudu)
I20260808 22:21:47.746968 9953 webserver.cc:460] Webserver started at http://127.9.176.254:35711/ using document root <none> and password file <none>
I20260808 22:21:47.747973 9953 fs_manager.cc:359] Metadata directory not provided
I20260808 22:21:47.748052 9953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:21:47.748482 9953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:21:47.751179 9953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/minicluster-data/master-0/data/instance:
uuid: "117ab7509e874353bc25ae52125a51a9"
format_stamp: "Formatted at 2026-08-08 22:21:47 on dist-test-slave-ncp9"
server_key: "fa68d96cd644564879cc79acf0193967"
server_key_iv: "56c625a4a2531eb38bfb94b011ee7002"
server_key_version: "encryptionkey@0"
I20260808 22:21:47.751727 9953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/minicluster-data/master-0/wal/instance:
uuid: "117ab7509e874353bc25ae52125a51a9"
format_stamp: "Formatted at 2026-08-08 22:21:47 on dist-test-slave-ncp9"
server_key: "fa68d96cd644564879cc79acf0193967"
server_key_iv: "56c625a4a2531eb38bfb94b011ee7002"
server_key_version: "encryptionkey@0"
I20260808 22:21:47.757431 9953 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260808 22:21:47.761291 9972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:21:47.763036 9953 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260808 22:21:47.763221 9953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/minicluster-data/master-0/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/minicluster-data/master-0/wal
uuid: "117ab7509e874353bc25ae52125a51a9"
format_stamp: "Formatted at 2026-08-08 22:21:47 on dist-test-slave-ncp9"
server_key: "fa68d96cd644564879cc79acf0193967"
server_key_iv: "56c625a4a2531eb38bfb94b011ee7002"
server_key_version: "encryptionkey@0"
I20260808 22:21:47.763334 9953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:21:47.787686 9953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:21:47.790895 9953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:21:47.791114 9953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:21:47.800670 9953 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.254:37449
I20260808 22:21:47.800721 10046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.254:37449 every 8 connection(s)
I20260808 22:21:47.802084 9953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
I20260808 22:21:47.804759 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 9953
I20260808 22:21:47.804957 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/minicluster-data/master-0/wal/instance
I20260808 22:21:47.806885 9923 external_mini_cluster.cc:1428] Setting key d042f346fc6e7c6253e65386da33134d
I20260808 22:21:47.808434 10047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:21:47.818058 10047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 117ab7509e874353bc25ae52125a51a9: Bootstrap starting.
I20260808 22:21:47.822000 10047 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 117ab7509e874353bc25ae52125a51a9: Neither blocks nor log segments found. Creating new log.
I20260808 22:21:47.823197 10047 log.cc:824] T 00000000000000000000000000000000 P 117ab7509e874353bc25ae52125a51a9: Log is configured to *not* fsync() on all Append() calls
I20260808 22:21:47.825868 10047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 117ab7509e874353bc25ae52125a51a9: No bootstrap required, opened a new log
Aug 08 22:21:47 dist-test-slave-ncp9 krb5kdc[9936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227707, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
I20260808 22:21:47.830088 10047 raft_consensus.cc:348] T 00000000000000000000000000000000 P 117ab7509e874353bc25ae52125a51a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "117ab7509e874353bc25ae52125a51a9" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 37449 } }
I20260808 22:21:47.830402 10047 raft_consensus.cc:374] T 00000000000000000000000000000000 P 117ab7509e874353bc25ae52125a51a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:21:47.830494 10047 raft_consensus.cc:729] T 00000000000000000000000000000000 P 117ab7509e874353bc25ae52125a51a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 117ab7509e874353bc25ae52125a51a9, State: Initialized, Role: FOLLOWER
I20260808 22:21:47.831300 10047 consensus_queue.cc:260] T 00000000000000000000000000000000 P 117ab7509e874353bc25ae52125a51a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "117ab7509e874353bc25ae52125a51a9" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 37449 } }
I20260808 22:21:47.831498 10047 raft_consensus.cc:388] T 00000000000000000000000000000000 P 117ab7509e874353bc25ae52125a51a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:21:47.831668 10047 raft_consensus.cc:482] T 00000000000000000000000000000000 P 117ab7509e874353bc25ae52125a51a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:21:47.831815 10047 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 117ab7509e874353bc25ae52125a51a9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:21:47.832717 10047 raft_consensus.cc:504] T 00000000000000000000000000000000 P 117ab7509e874353bc25ae52125a51a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "117ab7509e874353bc25ae52125a51a9" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 37449 } }
I20260808 22:21:47.833386 10047 leader_election.cc:302] T 00000000000000000000000000000000 P 117ab7509e874353bc25ae52125a51a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 117ab7509e874353bc25ae52125a51a9; no voters:
I20260808 22:21:47.833868 10047 leader_election.cc:288] T 00000000000000000000000000000000 P 117ab7509e874353bc25ae52125a51a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:21:47.833988 10055 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 117ab7509e874353bc25ae52125a51a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:21:47.834149 10052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:47.814724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34102 (local address 127.9.176.254:37449)
0808 22:21:47.815195 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0808 22:21:47.815214 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0808 22:21:47.815298 (+ 84us) server_negotiation.cc:407] Connection header received
0808 22:21:47.817679 (+ 2381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:21:47.817695 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:21:47.818096 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:21:47.818412 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:21:47.820732 (+ 2320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:47.821808 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:47.823043 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:47.823539 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:47.823687 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:47.827694 (+ 4007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:21:47.827727 (+ 33us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:21:47.827741 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:21:47.827769 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:21:47.830273 (+ 2504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:21:47.830857 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:21:47.830862 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:21:47.830866 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:21:47.830986 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:21:47.831308 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:21:47.831311 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:21:47.831313 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:21:47.831895 (+ 582us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:21:47.832101 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0808 22:21:47.832715 (+ 614us) server_negotiation.cc:299] Negotiation successful
0808 22:21:47.833028 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":128,"threads_started":1}
I20260808 22:21:47.834223 10055 raft_consensus.cc:686] T 00000000000000000000000000000000 P 117ab7509e874353bc25ae52125a51a9 [term 1 LEADER]: Becoming Leader. State: Replica: 117ab7509e874353bc25ae52125a51a9, State: Running, Role: LEADER
I20260808 22:21:47.834649 10055 consensus_queue.cc:237] T 00000000000000000000000000000000 P 117ab7509e874353bc25ae52125a51a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "117ab7509e874353bc25ae52125a51a9" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 37449 } }
I20260808 22:21:47.835768 10047 sys_catalog.cc:564] T 00000000000000000000000000000000 P 117ab7509e874353bc25ae52125a51a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:21:47.838455 10057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 117ab7509e874353bc25ae52125a51a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 117ab7509e874353bc25ae52125a51a9. Latest consensus state: current_term: 1 leader_uuid: "117ab7509e874353bc25ae52125a51a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "117ab7509e874353bc25ae52125a51a9" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 37449 } } }
I20260808 22:21:47.838593 10057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 117ab7509e874353bc25ae52125a51a9 [sys.catalog]: This master's current role is: LEADER
I20260808 22:21:47.839143 10056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 117ab7509e874353bc25ae52125a51a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "117ab7509e874353bc25ae52125a51a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "117ab7509e874353bc25ae52125a51a9" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 37449 } } }
I20260808 22:21:47.839478 10056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 117ab7509e874353bc25ae52125a51a9 [sys.catalog]: This master's current role is: LEADER
W20260808 22:21:47.843453 10076 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 117ab7509e874353bc25ae52125a51a9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 22:21:47.843601 10076 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 22:21:47.843688 10075 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:21:47.844597 10075 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:21:47.851511 10075 catalog_manager.cc:1347] Generated new cluster ID: a34140a654244d12aa7ad7e2e7dcaa2c
I20260808 22:21:47.851590 10075 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:21:47.859417 10075 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:21:47.860363 10075 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:21:47.865970 10075 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 117ab7509e874353bc25ae52125a51a9: Generated new TSK 0
I20260808 22:21:47.866883 10075 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.176.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.176.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.176.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.193.keytab.
Entry for principal kudu/127.9.176.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.193.keytab.
I20260808 22:21:47.897775 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.193:0
--local_ip_for_outbound_sockets=127.9.176.193
--webserver_interface=127.9.176.193
--webserver_port=0
--tserver_master_addrs=127.9.176.254:37449
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.193.keytab
--principal=kudu/127.9.176.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:36811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:21:48.017959 10085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:21:48.018230 10085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:21:48.018303 10085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:21:48.022310 10085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:21:48.022382 10085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:21:48.022470 10085 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.193
I20260808 22:21:48.027822 10085 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:36811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.193.keytab
--principal=kudu/127.9.176.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.176.193
--webserver_port=0
--tserver_master_addrs=127.9.176.254:37449
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10085
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.193
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:21:48.028981 10085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:21:48.030030 10085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:21:48.040315 10095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:21:48.040320 10096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:21:48.040315 10098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:21:48.040551 10085 server_base.cc:1034] running on GCE node
I20260808 22:21:48.041079 10085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:21:48.041721 10085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:21:48.042893 10085 hybrid_clock.cc:648] HybridClock initialized: now 1786227708042872 us; error 31 us; skew 500 ppm
Aug 08 22:21:48 dist-test-slave-ncp9 krb5kdc[9936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227708, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:21:48.045733 10085 init.cc:377] Logged in from keytab as kudu/127.9.176.193@KRBTEST.COM (short username kudu)
I20260808 22:21:48.046599 10085 webserver.cc:460] Webserver started at http://127.9.176.193:45785/ using document root <none> and password file <none>
I20260808 22:21:48.047137 10085 fs_manager.cc:359] Metadata directory not provided
I20260808 22:21:48.047204 10085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:21:48.047417 10085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:21:48.049528 10085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/minicluster-data/ts-0/data/instance:
uuid: "c73c01e0dcf8414194d9086a8ff94c2d"
format_stamp: "Formatted at 2026-08-08 22:21:48 on dist-test-slave-ncp9"
server_key: "87c81d94a35f5d672ca68abcd1248ee4"
server_key_iv: "0455103c8a6e7a7622d97ad95ed7dc59"
server_key_version: "encryptionkey@0"
I20260808 22:21:48.050086 10085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/minicluster-data/ts-0/wal/instance:
uuid: "c73c01e0dcf8414194d9086a8ff94c2d"
format_stamp: "Formatted at 2026-08-08 22:21:48 on dist-test-slave-ncp9"
server_key: "87c81d94a35f5d672ca68abcd1248ee4"
server_key_iv: "0455103c8a6e7a7622d97ad95ed7dc59"
server_key_version: "encryptionkey@0"
I20260808 22:21:48.053644 10085 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260808 22:21:48.055930 10108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:21:48.057147 10085 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 22:21:48.057250 10085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/minicluster-data/ts-0/wal
uuid: "c73c01e0dcf8414194d9086a8ff94c2d"
format_stamp: "Formatted at 2026-08-08 22:21:48 on dist-test-slave-ncp9"
server_key: "87c81d94a35f5d672ca68abcd1248ee4"
server_key_iv: "0455103c8a6e7a7622d97ad95ed7dc59"
server_key_version: "encryptionkey@0"
I20260808 22:21:48.057341 10085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:21:48.079855 10085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:21:48.082628 10085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:21:48.082799 10085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:21:48.083479 10085 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:21:48.084583 10085 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:21:48.084636 10085 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:21:48.084690 10085 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:21:48.084720 10085 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:21:48.098701 10085 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.193:45289
I20260808 22:21:48.098766 10276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.193:45289 every 8 connection(s)
I20260808 22:21:48.099610 10085 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
I20260808 22:21:48.105113 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 10085
I20260808 22:21:48.105217 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/minicluster-data/ts-0/wal/instance
I20260808 22:21:48.105492 9923 external_mini_cluster.cc:1428] Setting key ade237be8975774d068ca096fb0ea4ce
Aug 08 22:21:48 dist-test-slave-ncp9 krb5kdc[9936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227708, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.193@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
I20260808 22:21:48.115454 10052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:48.102112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.193:51005 (local address 127.9.176.254:37449)
0808 22:21:48.102299 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0808 22:21:48.102303 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:21:48.103156 (+ 853us) server_negotiation.cc:407] Connection header received
0808 22:21:48.104730 (+ 1574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:21:48.104733 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:21:48.104778 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:21:48.104850 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:21:48.106366 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:48.107103 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:48.108749 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:48.109045 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:48.109123 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:48.112036 (+ 2913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:21:48.112040 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:21:48.112042 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:21:48.112067 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:21:48.113405 (+ 1338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:21:48.113982 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:21:48.113985 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:21:48.113986 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:21:48.114061 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:21:48.114420 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:21:48.114422 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:21:48.114423 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:21:48.114587 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:21:48.114657 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0808 22:21:48.115239 (+ 582us) server_negotiation.cc:299] Negotiation successful
0808 22:21:48.115333 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":85}
I20260808 22:21:48.116647 10281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:48.102565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:37449 (local address 127.9.176.193:51005)
0808 22:21:48.103010 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0808 22:21:48.103052 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 22:21:48.104537 (+ 1485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:21:48.105078 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:21:48.105087 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:21:48.105578 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:21:48.106214 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:21:48.106233 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:21:48.107227 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:21:48.107231 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:21:48.108631 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:21:48.108643 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:21:48.109199 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:21:48.109203 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:21:48.109420 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:48.110028 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:21:48.110052 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:21:48.111859 (+ 1807us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:21:48.113511 (+ 1652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:21:48.113518 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:21:48.113535 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:21:48.113884 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:21:48.114147 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:21:48.114150 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:21:48.114152 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:21:48.114338 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:21:48.114680 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:21:48.114687 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:21:48.115005 (+ 318us) client_negotiation.cc:769] Sending connection context
0808 22:21:48.115223 (+ 218us) client_negotiation.cc:240] Negotiation successful
0808 22:21:48.115571 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":113,"threads_started":1}
WARNING: no policy specified for kudu/127.9.176.194@KRBTEST.COM; defaulting to no policy
I20260808 22:21:48.117843 10278 heartbeater.cc:347] Connected to a master server at 127.9.176.254:37449
I20260808 22:21:48.118075 10278 heartbeater.cc:464] Registering TS with master...
I20260808 22:21:48.118546 10278 heartbeater.cc:511] Master 127.9.176.254:37449 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.176.194@KRBTEST.COM" created.
I20260808 22:21:48.120549 9999 ts_manager.cc:194] Registered new tserver with Master: c73c01e0dcf8414194d9086a8ff94c2d (127.9.176.193:45289)
I20260808 22:21:48.122056 9999 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.176.193@KRBTEST.COM'} at 127.9.176.193:51005
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.176.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.194.keytab.
Entry for principal kudu/127.9.176.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.194.keytab.
I20260808 22:21:48.136029 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.194:0
--local_ip_for_outbound_sockets=127.9.176.194
--webserver_interface=127.9.176.194
--webserver_port=0
--tserver_master_addrs=127.9.176.254:37449
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.194.keytab
--principal=kudu/127.9.176.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:36811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:21:48.254247 10286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:21:48.254493 10286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:21:48.254535 10286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:21:48.258610 10286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:21:48.258674 10286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:21:48.258749 10286 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.194
I20260808 22:21:48.264142 10286 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:36811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.194.keytab
--principal=kudu/127.9.176.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.176.194
--webserver_port=0
--tserver_master_addrs=127.9.176.254:37449
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10286
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.194
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:21:48.265141 10286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:21:48.266117 10286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:21:48.275487 10294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:21:48.275496 10296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:21:48.275502 10293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:21:48.275648 10286 server_base.cc:1034] running on GCE node
I20260808 22:21:48.276180 10286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:21:48.276748 10286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:21:48.277896 10286 hybrid_clock.cc:648] HybridClock initialized: now 1786227708277875 us; error 57 us; skew 500 ppm
Aug 08 22:21:48 dist-test-slave-ncp9 krb5kdc[9936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227708, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:21:48.280527 10286 init.cc:377] Logged in from keytab as kudu/127.9.176.194@KRBTEST.COM (short username kudu)
I20260808 22:21:48.281368 10286 webserver.cc:460] Webserver started at http://127.9.176.194:45117/ using document root <none> and password file <none>
I20260808 22:21:48.281862 10286 fs_manager.cc:359] Metadata directory not provided
I20260808 22:21:48.281926 10286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:21:48.282116 10286 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:21:48.284054 10286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/minicluster-data/ts-1/data/instance:
uuid: "0992aee40c804d0fb8da8cb1499ac3b2"
format_stamp: "Formatted at 2026-08-08 22:21:48 on dist-test-slave-ncp9"
server_key: "9172cad70b4d2be58210aec5208ab35b"
server_key_iv: "da3e89b7ade6dc7ce7d984b9c15cd49a"
server_key_version: "encryptionkey@0"
I20260808 22:21:48.284574 10286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/minicluster-data/ts-1/wal/instance:
uuid: "0992aee40c804d0fb8da8cb1499ac3b2"
format_stamp: "Formatted at 2026-08-08 22:21:48 on dist-test-slave-ncp9"
server_key: "9172cad70b4d2be58210aec5208ab35b"
server_key_iv: "da3e89b7ade6dc7ce7d984b9c15cd49a"
server_key_version: "encryptionkey@0"
I20260808 22:21:48.288196 10286 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260808 22:21:48.290398 10306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:21:48.291507 10286 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 22:21:48.291607 10286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/minicluster-data/ts-1/wal
uuid: "0992aee40c804d0fb8da8cb1499ac3b2"
format_stamp: "Formatted at 2026-08-08 22:21:48 on dist-test-slave-ncp9"
server_key: "9172cad70b4d2be58210aec5208ab35b"
server_key_iv: "da3e89b7ade6dc7ce7d984b9c15cd49a"
server_key_version: "encryptionkey@0"
I20260808 22:21:48.291694 10286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:21:48.328325 10286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:21:48.331359 10286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:21:48.331540 10286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:21:48.332105 10286 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:21:48.332978 10286 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:21:48.333027 10286 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:21:48.333081 10286 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:21:48.333106 10286 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:21:48.349023 10286 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.194:40889
I20260808 22:21:48.349056 10470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.194:40889 every 8 connection(s)
I20260808 22:21:48.349872 10286 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
I20260808 22:21:48.353606 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 10286
I20260808 22:21:48.353684 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/minicluster-data/ts-1/wal/instance
I20260808 22:21:48.353890 9923 external_mini_cluster.cc:1428] Setting key bb58e0fd216701cfa83a84ef0aa09971
Aug 08 22:21:48 dist-test-slave-ncp9 krb5kdc[9936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227708, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.194@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
I20260808 22:21:48.363816 10052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:48.351823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.194:35087 (local address 127.9.176.254:37449)
0808 22:21:48.351944 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0808 22:21:48.351947 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:21:48.352801 (+ 854us) server_negotiation.cc:407] Connection header received
0808 22:21:48.354220 (+ 1419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:21:48.354223 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:21:48.354281 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:21:48.354348 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:21:48.355642 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:48.356165 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:48.357363 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:48.357583 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:48.357672 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:48.360255 (+ 2583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:21:48.360259 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:21:48.360263 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:21:48.360287 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:21:48.361790 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:21:48.362376 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:21:48.362378 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:21:48.362379 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:21:48.362448 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:21:48.362806 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:21:48.362808 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:21:48.362809 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:21:48.362968 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:21:48.363048 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0808 22:21:48.363610 (+ 562us) server_negotiation.cc:299] Negotiation successful
0808 22:21:48.363703 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260808 22:21:48.364996 10474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:48.352220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:37449 (local address 127.9.176.194:35087)
0808 22:21:48.352662 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0808 22:21:48.352705 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 22:21:48.354005 (+ 1300us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:21:48.354543 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:21:48.354552 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:21:48.355018 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:21:48.355505 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:21:48.355519 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:21:48.356279 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:21:48.356283 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:21:48.357249 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:21:48.357259 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:21:48.357720 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:21:48.357724 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:21:48.357894 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:48.358434 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:21:48.358450 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:21:48.360040 (+ 1590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:21:48.361903 (+ 1863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:21:48.361910 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:21:48.361926 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:21:48.362285 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:21:48.362525 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:21:48.362528 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:21:48.362530 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:21:48.362725 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:21:48.363070 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:21:48.363077 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:21:48.363375 (+ 298us) client_negotiation.cc:769] Sending connection context
0808 22:21:48.363582 (+ 207us) client_negotiation.cc:240] Negotiation successful
0808 22:21:48.363912 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":110,"threads_started":1}
WARNING: no policy specified for kudu/127.9.176.195@KRBTEST.COM; defaulting to no policy
I20260808 22:21:48.366114 10471 heartbeater.cc:347] Connected to a master server at 127.9.176.254:37449
I20260808 22:21:48.366349 10471 heartbeater.cc:464] Registering TS with master...
I20260808 22:21:48.366796 10471 heartbeater.cc:511] Master 127.9.176.254:37449 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.176.195@KRBTEST.COM" created.
I20260808 22:21:48.367890 9999 ts_manager.cc:194] Registered new tserver with Master: 0992aee40c804d0fb8da8cb1499ac3b2 (127.9.176.194:40889)
I20260808 22:21:48.368518 9999 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.176.194@KRBTEST.COM'} at 127.9.176.194:35087
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.176.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.195.keytab.
Entry for principal kudu/127.9.176.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.195.keytab.
I20260808 22:21:48.383950 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.195:0
--local_ip_for_outbound_sockets=127.9.176.195
--webserver_interface=127.9.176.195
--webserver_port=0
--tserver_master_addrs=127.9.176.254:37449
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.195.keytab
--principal=kudu/127.9.176.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:36811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:21:48.502815 10477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:21:48.503049 10477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:21:48.503091 10477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:21:48.506950 10477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:21:48.506996 10477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:21:48.507052 10477 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.195
I20260808 22:21:48.512012 10477 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:36811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.195.keytab
--principal=kudu/127.9.176.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.176.195
--webserver_port=0
--tserver_master_addrs=127.9.176.254:37449
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.195
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:21:48.512868 10477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:21:48.513815 10477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:21:48.522815 10487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:21:48.522910 10477 server_base.cc:1034] running on GCE node
W20260808 22:21:48.522801 10484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:21:48.522801 10485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:21:48.523419 10477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:21:48.523995 10477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:21:48.525180 10477 hybrid_clock.cc:648] HybridClock initialized: now 1786227708525161 us; error 30 us; skew 500 ppm
Aug 08 22:21:48 dist-test-slave-ncp9 krb5kdc[9936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227708, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:21:48.527885 10477 init.cc:377] Logged in from keytab as kudu/127.9.176.195@KRBTEST.COM (short username kudu)
I20260808 22:21:48.528673 10477 webserver.cc:460] Webserver started at http://127.9.176.195:34451/ using document root <none> and password file <none>
I20260808 22:21:48.529224 10477 fs_manager.cc:359] Metadata directory not provided
I20260808 22:21:48.529290 10477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:21:48.529489 10477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:21:48.531426 10477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/minicluster-data/ts-2/data/instance:
uuid: "4ae49795b6c9483494d6e67e55cea0e9"
format_stamp: "Formatted at 2026-08-08 22:21:48 on dist-test-slave-ncp9"
server_key: "11b0fcc5794dd4e19913a385fa4b51ae"
server_key_iv: "3990e6dfa3a9645b811b81a78130562f"
server_key_version: "encryptionkey@0"
I20260808 22:21:48.531955 10477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/minicluster-data/ts-2/wal/instance:
uuid: "4ae49795b6c9483494d6e67e55cea0e9"
format_stamp: "Formatted at 2026-08-08 22:21:48 on dist-test-slave-ncp9"
server_key: "11b0fcc5794dd4e19913a385fa4b51ae"
server_key_iv: "3990e6dfa3a9645b811b81a78130562f"
server_key_version: "encryptionkey@0"
I20260808 22:21:48.535449 10477 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260808 22:21:48.537751 10501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:21:48.538923 10477 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 22:21:48.539026 10477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/minicluster-data/ts-2/wal
uuid: "4ae49795b6c9483494d6e67e55cea0e9"
format_stamp: "Formatted at 2026-08-08 22:21:48 on dist-test-slave-ncp9"
server_key: "11b0fcc5794dd4e19913a385fa4b51ae"
server_key_iv: "3990e6dfa3a9645b811b81a78130562f"
server_key_version: "encryptionkey@0"
I20260808 22:21:48.539112 10477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:21:48.554329 10477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:21:48.557317 10477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:21:48.557480 10477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:21:48.558043 10477 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:21:48.558871 10477 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:21:48.558916 10477 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:21:48.558959 10477 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:21:48.558987 10477 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:21:48.573470 10477 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.195:34833
I20260808 22:21:48.573536 10664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.195:34833 every 8 connection(s)
I20260808 22:21:48.574327 10477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
I20260808 22:21:48.581897 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 10477
I20260808 22:21:48.582022 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786227706282457-9923-0/minicluster-data/ts-2/wal/instance
I20260808 22:21:48.582311 9923 external_mini_cluster.cc:1428] Setting key 3b9ad6ef5367fecbb33989afd0617b84
Aug 08 22:21:48 dist-test-slave-ncp9 krb5kdc[9936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227708, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.195@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
I20260808 22:21:48.587509 10052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:48.576308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.195:42403 (local address 127.9.176.254:37449)
0808 22:21:48.576449 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0808 22:21:48.576453 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:21:48.577848 (+ 1395us) server_negotiation.cc:407] Connection header received
0808 22:21:48.578666 (+ 818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:21:48.578669 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:21:48.578715 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:21:48.578786 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:21:48.580030 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:48.580554 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:48.581685 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:48.581994 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:48.582106 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:48.584761 (+ 2655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:21:48.584764 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:21:48.584766 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:21:48.584790 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:21:48.585939 (+ 1149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:21:48.586367 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:21:48.586369 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:21:48.586370 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:21:48.586435 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:21:48.586677 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:21:48.586679 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:21:48.586680 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:21:48.586836 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:21:48.586905 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0808 22:21:48.587300 (+ 395us) server_negotiation.cc:299] Negotiation successful
0808 22:21:48.587384 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 22:21:48.588274 10669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:48.576680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:37449 (local address 127.9.176.195:42403)
0808 22:21:48.577704 (+ 1024us) negotiation.cc:107] Waiting for socket to connect
0808 22:21:48.577745 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 22:21:48.578490 (+ 745us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:21:48.578933 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:21:48.578939 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:21:48.579345 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:21:48.579910 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:21:48.579928 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:21:48.580642 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:21:48.580645 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:21:48.581565 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:21:48.581575 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:21:48.582171 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:21:48.582175 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:21:48.582328 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:48.582910 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:21:48.582931 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:21:48.584601 (+ 1670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:21:48.586033 (+ 1432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:21:48.586038 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:21:48.586049 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:21:48.586294 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:21:48.586493 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:21:48.586495 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:21:48.586497 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:21:48.586615 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:21:48.586904 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:21:48.586908 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:21:48.587125 (+ 217us) client_negotiation.cc:769] Sending connection context
0808 22:21:48.587269 (+ 144us) client_negotiation.cc:240] Negotiation successful
0808 22:21:48.587492 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":839,"thread_start_us":706,"threads_started":1}
I20260808 22:21:48.589267 10665 heartbeater.cc:347] Connected to a master server at 127.9.176.254:37449
I20260808 22:21:48.589478 10665 heartbeater.cc:464] Registering TS with master...
I20260808 22:21:48.589920 10665 heartbeater.cc:511] Master 127.9.176.254:37449 requested a full tablet report, sending...
I20260808 22:21:48.590876 9999 ts_manager.cc:194] Registered new tserver with Master: 4ae49795b6c9483494d6e67e55cea0e9 (127.9.176.195:34833)
I20260808 22:21:48.591403 9999 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.176.195@KRBTEST.COM'} at 127.9.176.195:42403
I20260808 22:21:48.596626 9923 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 22:21:48 dist-test-slave-ncp9 krb5kdc[9936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227708, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 22:21:48 dist-test-slave-ncp9 krb5kdc[9936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227708, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.176.193@KRBTEST.COM
I20260808 22:21:48.625757 10678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:48.616810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33914 (local address 127.9.176.193:45289)
0808 22:21:48.617123 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0808 22:21:48.617141 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0808 22:21:48.617175 (+ 34us) server_negotiation.cc:407] Connection header received
0808 22:21:48.617307 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:21:48.617314 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:21:48.617506 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:21:48.617666 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:21:48.618346 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:48.619110 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:48.619932 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:48.620305 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:48.620403 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:48.622672 (+ 2269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:21:48.622681 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:21:48.622693 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:21:48.622715 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:21:48.623982 (+ 1267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:21:48.624389 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:21:48.624394 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:21:48.624395 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:21:48.624475 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:21:48.624727 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:21:48.624731 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:21:48.624732 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:21:48.625016 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:21:48.625216 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0808 22:21:48.625434 (+ 218us) server_negotiation.cc:299] Negotiation successful
0808 22:21:48.625599 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":119,"threads_started":1}
W20260808 22:21:48.626794 10151 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33914
Aug 08 22:21:48 dist-test-slave-ncp9 krb5kdc[9936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227708, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 22:21:48 dist-test-slave-ncp9 krb5kdc[9936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227708, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.176.193@KRBTEST.COM
I20260808 22:21:48.654583 10678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:48.646882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33920 (local address 127.9.176.193:45289)
0808 22:21:48.647044 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0808 22:21:48.647048 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:21:48.647106 (+ 58us) server_negotiation.cc:407] Connection header received
0808 22:21:48.647229 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:21:48.647232 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:21:48.647278 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:21:48.647346 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:21:48.647977 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:48.648489 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:48.649320 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:48.649547 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:48.649632 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:48.651768 (+ 2136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:21:48.651771 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:21:48.651773 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:21:48.651796 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:21:48.653176 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:21:48.653577 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:21:48.653579 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:21:48.653580 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:21:48.653659 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:21:48.653844 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:21:48.653846 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:21:48.653847 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:21:48.653992 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:21:48.654095 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0808 22:21:48.654333 (+ 238us) server_negotiation.cc:299] Negotiation successful
0808 22:21:48.654425 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260808 22:21:48.655670 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 10085
I20260808 22:21:48.675513 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 10286
I20260808 22:21:48.683774 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 10477
I20260808 22:21:48.692332 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 9953
2026-08-08T22:21:48Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2382 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 22:21:48 dist-test-slave-ncp9 krb5kdc[10693](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 22:21:48 dist-test-slave-ncp9 krb5kdc[10693](info): set up 2 sockets
Aug 08 22:21:48 dist-test-slave-ncp9 krb5kdc[10693](info): commencing operation
krb5kdc: starting...
W20260808 22:21:49.817178 9923 mini_kdc.cc:121] Time spent starting KDC: real 1.100s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/krb5kdc/test-user.keytab.
Aug 08 22:21:49 dist-test-slave-ncp9 krb5kdc[10693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227709, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T22:21:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:21:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.176.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.176.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.176.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab.
Entry for principal kudu/127.9.176.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab.
I20260808 22:21:49.931550 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:44265
--webserver_interface=127.9.176.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab
--principal=kudu/127.9.176.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:38773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.176.254:44265
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:21:50.041018 10710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:21:50.041278 10710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:21:50.041319 10710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:21:50.045257 10710 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:21:50.045316 10710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:21:50.045341 10710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:21:50.045378 10710 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:21:50.045413 10710 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:21:50.050585 10710 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:38773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.176.254:44265
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab
--principal=kudu/127.9.176.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:44265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.176.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10710
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:21:50.051642 10710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:21:50.052659 10710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:21:50.060767 10723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:21:50.060767 10721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:21:50.060793 10720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:21:50.061172 10710 server_base.cc:1034] running on GCE node
I20260808 22:21:50.061707 10710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:21:50.062765 10710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:21:50.063941 10710 hybrid_clock.cc:648] HybridClock initialized: now 1786227710063923 us; error 31 us; skew 500 ppm
Aug 08 22:21:50 dist-test-slave-ncp9 krb5kdc[10693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227710, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:21:50.066612 10710 init.cc:377] Logged in from keytab as kudu/127.9.176.254@KRBTEST.COM (short username kudu)
I20260808 22:21:50.067476 10710 webserver.cc:460] Webserver started at http://127.9.176.254:46589/ using document root <none> and password file <none>
I20260808 22:21:50.067958 10710 fs_manager.cc:359] Metadata directory not provided
I20260808 22:21:50.068015 10710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:21:50.068193 10710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:21:50.070114 10710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/minicluster-data/master-0/data/instance:
uuid: "321f7bba5c1345ff9627f505e3d337ee"
format_stamp: "Formatted at 2026-08-08 22:21:50 on dist-test-slave-ncp9"
server_key: "0cc68134f0bcf4ff2052a5b4e3f7f198"
server_key_iv: "8acedcf44a002ac647d75c5c11d22007"
server_key_version: "encryptionkey@0"
I20260808 22:21:50.070629 10710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/minicluster-data/master-0/wal/instance:
uuid: "321f7bba5c1345ff9627f505e3d337ee"
format_stamp: "Formatted at 2026-08-08 22:21:50 on dist-test-slave-ncp9"
server_key: "0cc68134f0bcf4ff2052a5b4e3f7f198"
server_key_iv: "8acedcf44a002ac647d75c5c11d22007"
server_key_version: "encryptionkey@0"
I20260808 22:21:50.074126 10710 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 22:21:50.076319 10730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:21:50.077504 10710 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 22:21:50.077605 10710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/minicluster-data/master-0/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/minicluster-data/master-0/wal
uuid: "321f7bba5c1345ff9627f505e3d337ee"
format_stamp: "Formatted at 2026-08-08 22:21:50 on dist-test-slave-ncp9"
server_key: "0cc68134f0bcf4ff2052a5b4e3f7f198"
server_key_iv: "8acedcf44a002ac647d75c5c11d22007"
server_key_version: "encryptionkey@0"
I20260808 22:21:50.077688 10710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:21:50.085942 10710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:21:50.088421 10710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:21:50.088568 10710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:21:50.094942 10710 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.254:44265
I20260808 22:21:50.094981 10815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.254:44265 every 8 connection(s)
I20260808 22:21:50.095836 10710 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
I20260808 22:21:50.096550 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 10710
I20260808 22:21:50.096639 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/minicluster-data/master-0/wal/instance
I20260808 22:21:50.096866 9923 external_mini_cluster.cc:1428] Setting key 26ecab1eda96ded50a788f9ec9dddbb2
I20260808 22:21:50.098970 10816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:21:50.104373 10816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 321f7bba5c1345ff9627f505e3d337ee: Bootstrap starting.
Aug 08 22:21:50 dist-test-slave-ncp9 krb5kdc[10693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227709, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
I20260808 22:21:50.106516 10816 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 321f7bba5c1345ff9627f505e3d337ee: Neither blocks nor log segments found. Creating new log.
I20260808 22:21:50.107203 10816 log.cc:824] T 00000000000000000000000000000000 P 321f7bba5c1345ff9627f505e3d337ee: Log is configured to *not* fsync() on all Append() calls
I20260808 22:21:50.108983 10816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 321f7bba5c1345ff9627f505e3d337ee: No bootstrap required, opened a new log
I20260808 22:21:50.110256 10819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:50.098421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37236 (local address 127.9.176.254:44265)
0808 22:21:50.098857 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0808 22:21:50.098872 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 22:21:50.098906 (+ 34us) server_negotiation.cc:407] Connection header received
0808 22:21:50.099541 (+ 635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:21:50.099558 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:21:50.099884 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:21:50.100221 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:21:50.100981 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:50.102041 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:50.102880 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:50.103339 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:50.103481 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:50.105858 (+ 2377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:21:50.105863 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:21:50.105874 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:21:50.105902 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:21:50.107464 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:21:50.107866 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:21:50.107871 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:21:50.107875 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:21:50.107984 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:21:50.108250 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:21:50.108253 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:21:50.108254 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:21:50.108642 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:21:50.108814 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0808 22:21:50.109111 (+ 297us) server_negotiation.cc:299] Negotiation successful
0808 22:21:50.109383 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":143,"threads_started":1}
I20260808 22:21:50.112115 10816 raft_consensus.cc:348] T 00000000000000000000000000000000 P 321f7bba5c1345ff9627f505e3d337ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "321f7bba5c1345ff9627f505e3d337ee" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 44265 } }
I20260808 22:21:50.112393 10816 raft_consensus.cc:374] T 00000000000000000000000000000000 P 321f7bba5c1345ff9627f505e3d337ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:21:50.112458 10816 raft_consensus.cc:729] T 00000000000000000000000000000000 P 321f7bba5c1345ff9627f505e3d337ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 321f7bba5c1345ff9627f505e3d337ee, State: Initialized, Role: FOLLOWER
I20260808 22:21:50.112836 10816 consensus_queue.cc:260] T 00000000000000000000000000000000 P 321f7bba5c1345ff9627f505e3d337ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "321f7bba5c1345ff9627f505e3d337ee" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 44265 } }
I20260808 22:21:50.112986 10816 raft_consensus.cc:388] T 00000000000000000000000000000000 P 321f7bba5c1345ff9627f505e3d337ee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:21:50.113036 10816 raft_consensus.cc:482] T 00000000000000000000000000000000 P 321f7bba5c1345ff9627f505e3d337ee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:21:50.113106 10816 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 321f7bba5c1345ff9627f505e3d337ee [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:21:50.114017 10816 raft_consensus.cc:504] T 00000000000000000000000000000000 P 321f7bba5c1345ff9627f505e3d337ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "321f7bba5c1345ff9627f505e3d337ee" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 44265 } }
I20260808 22:21:50.114367 10816 leader_election.cc:302] T 00000000000000000000000000000000 P 321f7bba5c1345ff9627f505e3d337ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 321f7bba5c1345ff9627f505e3d337ee; no voters:
I20260808 22:21:50.114666 10816 leader_election.cc:288] T 00000000000000000000000000000000 P 321f7bba5c1345ff9627f505e3d337ee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:21:50.114745 10821 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 321f7bba5c1345ff9627f505e3d337ee [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:21:50.114936 10821 raft_consensus.cc:686] T 00000000000000000000000000000000 P 321f7bba5c1345ff9627f505e3d337ee [term 1 LEADER]: Becoming Leader. State: Replica: 321f7bba5c1345ff9627f505e3d337ee, State: Running, Role: LEADER
I20260808 22:21:50.115247 10821 consensus_queue.cc:237] T 00000000000000000000000000000000 P 321f7bba5c1345ff9627f505e3d337ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "321f7bba5c1345ff9627f505e3d337ee" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 44265 } }
I20260808 22:21:50.115777 10816 sys_catalog.cc:564] T 00000000000000000000000000000000 P 321f7bba5c1345ff9627f505e3d337ee [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:21:50.116739 10827 sys_catalog.cc:455] T 00000000000000000000000000000000 P 321f7bba5c1345ff9627f505e3d337ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 321f7bba5c1345ff9627f505e3d337ee. Latest consensus state: current_term: 1 leader_uuid: "321f7bba5c1345ff9627f505e3d337ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "321f7bba5c1345ff9627f505e3d337ee" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 44265 } } }
I20260808 22:21:50.116873 10827 sys_catalog.cc:458] T 00000000000000000000000000000000 P 321f7bba5c1345ff9627f505e3d337ee [sys.catalog]: This master's current role is: LEADER
I20260808 22:21:50.117182 10826 sys_catalog.cc:455] T 00000000000000000000000000000000 P 321f7bba5c1345ff9627f505e3d337ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "321f7bba5c1345ff9627f505e3d337ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "321f7bba5c1345ff9627f505e3d337ee" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 44265 } } }
I20260808 22:21:50.117278 10826 sys_catalog.cc:458] T 00000000000000000000000000000000 P 321f7bba5c1345ff9627f505e3d337ee [sys.catalog]: This master's current role is: LEADER
I20260808 22:21:50.117978 10837 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:21:50.120500 10837 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:21:50.125664 10837 catalog_manager.cc:1347] Generated new cluster ID: 5395b38b8f7e454aaa1fbefb003dcfc8
I20260808 22:21:50.125732 10837 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:21:50.137192 10837 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:21:50.137998 10837 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:21:50.144538 10837 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 321f7bba5c1345ff9627f505e3d337ee: Generated new TSK 0
I20260808 22:21:50.145179 10837 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.176.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.176.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.176.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.193.keytab.
Entry for principal kudu/127.9.176.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.193.keytab.
I20260808 22:21:50.173326 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.193:0
--local_ip_for_outbound_sockets=127.9.176.193
--webserver_interface=127.9.176.193
--webserver_port=0
--tserver_master_addrs=127.9.176.254:44265
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.193.keytab
--principal=kudu/127.9.176.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:38773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:21:50.282958 10852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:21:50.283190 10852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:21:50.283236 10852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:21:50.287102 10852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:21:50.287151 10852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:21:50.287220 10852 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.193
I20260808 22:21:50.292173 10852 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:38773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.193.keytab
--principal=kudu/127.9.176.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.176.193
--webserver_port=0
--tserver_master_addrs=127.9.176.254:44265
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10852
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.193
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:21:50.293131 10852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:21:50.294072 10852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:21:50.303210 10858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:21:50.303210 10859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:21:50.303354 10852 server_base.cc:1034] running on GCE node
W20260808 22:21:50.303261 10862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:21:50.303817 10852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:21:50.304397 10852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:21:50.305548 10852 hybrid_clock.cc:648] HybridClock initialized: now 1786227710305535 us; error 24 us; skew 500 ppm
Aug 08 22:21:50 dist-test-slave-ncp9 krb5kdc[10693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227710, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:21:50.308274 10852 init.cc:377] Logged in from keytab as kudu/127.9.176.193@KRBTEST.COM (short username kudu)
I20260808 22:21:50.309185 10852 webserver.cc:460] Webserver started at http://127.9.176.193:46807/ using document root <none> and password file <none>
I20260808 22:21:50.309710 10852 fs_manager.cc:359] Metadata directory not provided
I20260808 22:21:50.309777 10852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:21:50.309959 10852 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:21:50.311877 10852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/minicluster-data/ts-0/data/instance:
uuid: "19d8135fb1aa4ed184b4960acd4c4eea"
format_stamp: "Formatted at 2026-08-08 22:21:50 on dist-test-slave-ncp9"
server_key: "8706efac1209e83bba0f02661bc98072"
server_key_iv: "f4a8847a799bbdd040f88dc46788338e"
server_key_version: "encryptionkey@0"
I20260808 22:21:50.312395 10852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/minicluster-data/ts-0/wal/instance:
uuid: "19d8135fb1aa4ed184b4960acd4c4eea"
format_stamp: "Formatted at 2026-08-08 22:21:50 on dist-test-slave-ncp9"
server_key: "8706efac1209e83bba0f02661bc98072"
server_key_iv: "f4a8847a799bbdd040f88dc46788338e"
server_key_version: "encryptionkey@0"
I20260808 22:21:50.315865 10852 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260808 22:21:50.318224 10871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:21:50.319366 10852 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 22:21:50.319473 10852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/minicluster-data/ts-0/wal
uuid: "19d8135fb1aa4ed184b4960acd4c4eea"
format_stamp: "Formatted at 2026-08-08 22:21:50 on dist-test-slave-ncp9"
server_key: "8706efac1209e83bba0f02661bc98072"
server_key_iv: "f4a8847a799bbdd040f88dc46788338e"
server_key_version: "encryptionkey@0"
I20260808 22:21:50.319564 10852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:21:50.350001 10852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:21:50.353099 10852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:21:50.353276 10852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:21:50.353847 10852 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:21:50.354676 10852 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:21:50.354723 10852 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:21:50.354776 10852 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:21:50.354804 10852 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:21:50.362939 10852 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.193:33471
I20260808 22:21:50.362968 11046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.193:33471 every 8 connection(s)
I20260808 22:21:50.363780 10852 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
I20260808 22:21:50.369155 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 10852
I20260808 22:21:50.369282 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/minicluster-data/ts-0/wal/instance
I20260808 22:21:50.369532 9923 external_mini_cluster.cc:1428] Setting key ad2cc5863823c2119025284c31e3aa58
Aug 08 22:21:50 dist-test-slave-ncp9 krb5kdc[10693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227710, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.193@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
I20260808 22:21:50.377264 10819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:50.365508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.193:51179 (local address 127.9.176.254:44265)
0808 22:21:50.365649 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0808 22:21:50.365653 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:21:50.366416 (+ 763us) server_negotiation.cc:407] Connection header received
0808 22:21:50.367674 (+ 1258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:21:50.367677 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:21:50.367735 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:21:50.367807 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:21:50.369007 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:50.369776 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:50.371052 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:50.371378 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:50.371481 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:50.373822 (+ 2341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:21:50.373826 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:21:50.373829 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:21:50.373857 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:21:50.375131 (+ 1274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:21:50.375753 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:21:50.375757 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:21:50.375759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:21:50.375838 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:21:50.376152 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:21:50.376156 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:21:50.376157 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:21:50.376331 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:21:50.376439 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0808 22:21:50.376969 (+ 530us) server_negotiation.cc:299] Negotiation successful
0808 22:21:50.377107 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 22:21:50.378005 11055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:50.365847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:44265 (local address 127.9.176.193:51179)
0808 22:21:50.366276 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0808 22:21:50.366319 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 22:21:50.367465 (+ 1146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:21:50.367991 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:21:50.367997 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:21:50.368341 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:21:50.368801 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:21:50.368815 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:21:50.369910 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:21:50.369914 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:21:50.370934 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:21:50.370944 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:21:50.371504 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:21:50.371506 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:21:50.371647 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:50.372152 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:21:50.372169 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:21:50.373651 (+ 1482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:21:50.375343 (+ 1692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:21:50.375349 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:21:50.375363 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:21:50.375652 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:21:50.375922 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:21:50.375926 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:21:50.375927 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:21:50.376061 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:21:50.376454 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:21:50.376461 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:21:50.376710 (+ 249us) client_negotiation.cc:769] Sending connection context
0808 22:21:50.376875 (+ 165us) client_negotiation.cc:240] Negotiation successful
0808 22:21:50.377200 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":109,"threads_started":1}
I20260808 22:21:50.379143 11047 heartbeater.cc:347] Connected to a master server at 127.9.176.254:44265
WARNING: no policy specified for kudu/127.9.176.194@KRBTEST.COM; defaulting to no policy
I20260808 22:21:50.379355 11047 heartbeater.cc:464] Registering TS with master...
I20260808 22:21:50.379812 11047 heartbeater.cc:511] Master 127.9.176.254:44265 requested a full tablet report, sending...
I20260808 22:21:50.381304 10762 ts_manager.cc:194] Registered new tserver with Master: 19d8135fb1aa4ed184b4960acd4c4eea (127.9.176.193:33471)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.176.194@KRBTEST.COM" created.
I20260808 22:21:50.382637 10762 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.176.193@KRBTEST.COM'} at 127.9.176.193:51179
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.176.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.194.keytab.
Entry for principal kudu/127.9.176.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.194.keytab.
I20260808 22:21:50.395364 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.194:0
--local_ip_for_outbound_sockets=127.9.176.194
--webserver_interface=127.9.176.194
--webserver_port=0
--tserver_master_addrs=127.9.176.254:44265
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.194.keytab
--principal=kudu/127.9.176.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:38773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:21:50.502408 11062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:21:50.502739 11062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:21:50.502804 11062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:21:50.506759 11062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:21:50.506824 11062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:21:50.506886 11062 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.194
I20260808 22:21:50.512172 11062 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:38773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.194.keytab
--principal=kudu/127.9.176.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.176.194
--webserver_port=0
--tserver_master_addrs=127.9.176.254:44265
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11062
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.194
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:21:50.513278 11062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:21:50.514294 11062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:21:50.523895 11072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:21:50.523857 11082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:21:50.523840 11073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:21:50.524302 11062 server_base.cc:1034] running on GCE node
I20260808 22:21:50.524704 11062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:21:50.525334 11062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:21:50.526489 11062 hybrid_clock.cc:648] HybridClock initialized: now 1786227710526475 us; error 27 us; skew 500 ppm
Aug 08 22:21:50 dist-test-slave-ncp9 krb5kdc[10693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227710, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:21:50.529160 11062 init.cc:377] Logged in from keytab as kudu/127.9.176.194@KRBTEST.COM (short username kudu)
I20260808 22:21:50.529992 11062 webserver.cc:460] Webserver started at http://127.9.176.194:40569/ using document root <none> and password file <none>
I20260808 22:21:50.530673 11062 fs_manager.cc:359] Metadata directory not provided
I20260808 22:21:50.530752 11062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:21:50.530949 11062 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:21:50.532941 11062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/minicluster-data/ts-1/data/instance:
uuid: "27a4e31052d24e93b13a7417a967ebec"
format_stamp: "Formatted at 2026-08-08 22:21:50 on dist-test-slave-ncp9"
server_key: "f0efcae71810b207adebbb06eb7a0d97"
server_key_iv: "35ce9291e592c9ce0c0a1e9d7d93ff5e"
server_key_version: "encryptionkey@0"
I20260808 22:21:50.533504 11062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/minicluster-data/ts-1/wal/instance:
uuid: "27a4e31052d24e93b13a7417a967ebec"
format_stamp: "Formatted at 2026-08-08 22:21:50 on dist-test-slave-ncp9"
server_key: "f0efcae71810b207adebbb06eb7a0d97"
server_key_iv: "35ce9291e592c9ce0c0a1e9d7d93ff5e"
server_key_version: "encryptionkey@0"
I20260808 22:21:50.536964 11062 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 22:21:50.539165 11092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:21:50.540254 11062 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 22:21:50.540349 11062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/minicluster-data/ts-1/wal
uuid: "27a4e31052d24e93b13a7417a967ebec"
format_stamp: "Formatted at 2026-08-08 22:21:50 on dist-test-slave-ncp9"
server_key: "f0efcae71810b207adebbb06eb7a0d97"
server_key_iv: "35ce9291e592c9ce0c0a1e9d7d93ff5e"
server_key_version: "encryptionkey@0"
I20260808 22:21:50.540442 11062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:21:50.564198 11062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:21:50.567209 11062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:21:50.567375 11062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:21:50.567934 11062 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:21:50.568733 11062 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:21:50.568780 11062 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:21:50.568823 11062 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:21:50.568851 11062 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:21:50.577328 11062 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.194:43515
I20260808 22:21:50.577369 11257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.194:43515 every 8 connection(s)
I20260808 22:21:50.578168 11062 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
I20260808 22:21:50.581131 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 11062
I20260808 22:21:50.581233 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/minicluster-data/ts-1/wal/instance
I20260808 22:21:50.581478 9923 external_mini_cluster.cc:1428] Setting key dac5e0cd323a982d87c1912cc15027bd
Aug 08 22:21:50 dist-test-slave-ncp9 krb5kdc[10693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227710, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.194@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.176.195@KRBTEST.COM; defaulting to no policy
I20260808 22:21:50.592242 10819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:50.579907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.194:48291 (local address 127.9.176.254:44265)
0808 22:21:50.580030 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0808 22:21:50.580034 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:21:50.580818 (+ 784us) server_negotiation.cc:407] Connection header received
0808 22:21:50.582313 (+ 1495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:21:50.582317 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:21:50.582371 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:21:50.582443 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:21:50.583628 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:50.584147 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:50.585766 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:50.586051 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:50.586155 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:50.588709 (+ 2554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:21:50.588712 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:21:50.588714 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:21:50.588740 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:21:50.590064 (+ 1324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:21:50.590686 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:21:50.590690 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:21:50.590692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:21:50.590774 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:21:50.591182 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:21:50.591186 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:21:50.591187 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:21:50.591412 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:21:50.591498 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0808 22:21:50.591965 (+ 467us) server_negotiation.cc:299] Negotiation successful
0808 22:21:50.592068 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260808 22:21:50.593104 11261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:50.580242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:44265 (local address 127.9.176.194:48291)
0808 22:21:50.580688 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0808 22:21:50.580730 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 22:21:50.582097 (+ 1367us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:21:50.582617 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:21:50.582626 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:21:50.583017 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:21:50.583487 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:21:50.583502 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:21:50.584263 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:21:50.584267 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:21:50.585633 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:21:50.585646 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:21:50.586191 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:21:50.586194 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:21:50.586331 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:50.586858 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:21:50.586874 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:21:50.588539 (+ 1665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:21:50.590183 (+ 1644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:21:50.590189 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:21:50.590206 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:21:50.590574 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:21:50.590867 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:21:50.590871 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:21:50.590873 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:21:50.591059 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:21:50.591519 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:21:50.591526 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:21:50.591767 (+ 241us) client_negotiation.cc:769] Sending connection context
0808 22:21:50.591925 (+ 158us) client_negotiation.cc:240] Negotiation successful
0808 22:21:50.592197 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":132,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.176.195@KRBTEST.COM" created.
I20260808 22:21:50.595566 11258 heartbeater.cc:347] Connected to a master server at 127.9.176.254:44265
I20260808 22:21:50.595810 11258 heartbeater.cc:464] Registering TS with master...
I20260808 22:21:50.596293 11258 heartbeater.cc:511] Master 127.9.176.254:44265 requested a full tablet report, sending...
I20260808 22:21:50.597350 10762 ts_manager.cc:194] Registered new tserver with Master: 27a4e31052d24e93b13a7417a967ebec (127.9.176.194:43515)
I20260808 22:21:50.597898 10762 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.176.194@KRBTEST.COM'} at 127.9.176.194:48291
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.176.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.195.keytab.
Entry for principal kudu/127.9.176.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.195.keytab.
I20260808 22:21:50.607775 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.195:0
--local_ip_for_outbound_sockets=127.9.176.195
--webserver_interface=127.9.176.195
--webserver_port=0
--tserver_master_addrs=127.9.176.254:44265
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.195.keytab
--principal=kudu/127.9.176.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:38773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:21:50.719550 11266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:21:50.719825 11266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:21:50.719877 11266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:21:50.724047 11266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:21:50.724109 11266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:21:50.724166 11266 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.195
I20260808 22:21:50.729339 11266 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:38773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.195.keytab
--principal=kudu/127.9.176.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.176.195
--webserver_port=0
--tserver_master_addrs=127.9.176.254:44265
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11266
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.195
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:21:50.730314 11266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:21:50.731248 11266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:21:50.740350 11278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:21:50.740459 11276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:21:50.740470 11273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:21:50.740504 11266 server_base.cc:1034] running on GCE node
I20260808 22:21:50.740994 11266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:21:50.741569 11266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:21:50.742733 11266 hybrid_clock.cc:648] HybridClock initialized: now 1786227710742719 us; error 32 us; skew 500 ppm
Aug 08 22:21:50 dist-test-slave-ncp9 krb5kdc[10693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227710, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:21:50.745379 11266 init.cc:377] Logged in from keytab as kudu/127.9.176.195@KRBTEST.COM (short username kudu)
I20260808 22:21:50.746177 11266 webserver.cc:460] Webserver started at http://127.9.176.195:41099/ using document root <none> and password file <none>
I20260808 22:21:50.746693 11266 fs_manager.cc:359] Metadata directory not provided
I20260808 22:21:50.746757 11266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:21:50.746945 11266 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:21:50.748872 11266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/minicluster-data/ts-2/data/instance:
uuid: "9e3683d2459a4794b112a81f78cec799"
format_stamp: "Formatted at 2026-08-08 22:21:50 on dist-test-slave-ncp9"
server_key: "d1d794aec5e758996e15301fae93a48a"
server_key_iv: "e6c6e1018042444bf4f125234156eafa"
server_key_version: "encryptionkey@0"
I20260808 22:21:50.749439 11266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/minicluster-data/ts-2/wal/instance:
uuid: "9e3683d2459a4794b112a81f78cec799"
format_stamp: "Formatted at 2026-08-08 22:21:50 on dist-test-slave-ncp9"
server_key: "d1d794aec5e758996e15301fae93a48a"
server_key_iv: "e6c6e1018042444bf4f125234156eafa"
server_key_version: "encryptionkey@0"
I20260808 22:21:50.753000 11266 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260808 22:21:50.755187 11289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:21:50.756276 11266 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 22:21:50.756382 11266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/minicluster-data/ts-2/wal
uuid: "9e3683d2459a4794b112a81f78cec799"
format_stamp: "Formatted at 2026-08-08 22:21:50 on dist-test-slave-ncp9"
server_key: "d1d794aec5e758996e15301fae93a48a"
server_key_iv: "e6c6e1018042444bf4f125234156eafa"
server_key_version: "encryptionkey@0"
I20260808 22:21:50.756460 11266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:21:50.773022 11266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:21:50.776038 11266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:21:50.776189 11266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:21:50.776712 11266 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:21:50.777540 11266 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:21:50.777588 11266 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:21:50.777635 11266 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:21:50.777657 11266 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:21:50.787109 11266 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.195:38799
I20260808 22:21:50.787137 11455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.195:38799 every 8 connection(s)
I20260808 22:21:50.787940 11266 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
I20260808 22:21:50.793771 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 11266
I20260808 22:21:50.793886 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786227706282457-9923-0/minicluster-data/ts-2/wal/instance
I20260808 22:21:50.794145 9923 external_mini_cluster.cc:1428] Setting key fbfdbe84efcd72b3443f1a3584b98ea0
Aug 08 22:21:50 dist-test-slave-ncp9 krb5kdc[10693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227710, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.195@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
I20260808 22:21:50.801712 10819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:50.789731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.195:43781 (local address 127.9.176.254:44265)
0808 22:21:50.789863 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0808 22:21:50.789867 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:21:50.790730 (+ 863us) server_negotiation.cc:407] Connection header received
0808 22:21:50.792066 (+ 1336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:21:50.792069 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:21:50.792128 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:21:50.792200 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:21:50.793575 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:50.794320 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:50.795644 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:50.795957 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:50.796052 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:50.798342 (+ 2290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:21:50.798345 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:21:50.798347 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:21:50.798370 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:21:50.799595 (+ 1225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:21:50.800059 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:21:50.800061 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:21:50.800063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:21:50.800140 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:21:50.800597 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:21:50.800601 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:21:50.800603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:21:50.800792 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:21:50.800885 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0808 22:21:50.801431 (+ 546us) server_negotiation.cc:299] Negotiation successful
0808 22:21:50.801556 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 22:21:50.802444 11459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:50.790076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:44265 (local address 127.9.176.195:43781)
0808 22:21:50.790586 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0808 22:21:50.790632 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0808 22:21:50.791855 (+ 1223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:21:50.792381 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:21:50.792388 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:21:50.792876 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:21:50.793429 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:21:50.793445 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:21:50.794454 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:21:50.794458 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:21:50.795530 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:21:50.795541 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:21:50.796099 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:21:50.796103 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:21:50.796282 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:50.796794 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:21:50.796810 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:21:50.798194 (+ 1384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:21:50.799710 (+ 1516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:21:50.799715 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:21:50.799729 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:21:50.799965 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:21:50.800224 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:21:50.800227 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:21:50.800229 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:21:50.800455 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:21:50.800932 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:21:50.800939 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:21:50.801163 (+ 224us) client_negotiation.cc:769] Sending connection context
0808 22:21:50.801357 (+ 194us) client_negotiation.cc:240] Negotiation successful
0808 22:21:50.801669 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":124,"threads_started":1}
I20260808 22:21:50.803480 11456 heartbeater.cc:347] Connected to a master server at 127.9.176.254:44265
I20260808 22:21:50.803699 11456 heartbeater.cc:464] Registering TS with master...
I20260808 22:21:50.804133 11456 heartbeater.cc:511] Master 127.9.176.254:44265 requested a full tablet report, sending...
I20260808 22:21:50.805102 10762 ts_manager.cc:194] Registered new tserver with Master: 9e3683d2459a4794b112a81f78cec799 (127.9.176.195:38799)
I20260808 22:21:50.805661 10762 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.176.195@KRBTEST.COM'} at 127.9.176.195:43781
I20260808 22:21:50.807663 9923 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 22:21:50.818305 10819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:50.811658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37240 (local address 127.9.176.254:44265)
0808 22:21:50.811791 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0808 22:21:50.811795 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:21:50.811975 (+ 180us) server_negotiation.cc:407] Connection header received
0808 22:21:50.812070 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:21:50.812072 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:21:50.812108 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:21:50.812168 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:21:50.812805 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:50.813331 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:50.814090 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:50.814294 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:50.814356 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:50.815542 (+ 1186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:21:50.815545 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:21:50.815546 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:21:50.815561 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:21:50.816824 (+ 1263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:21:50.817283 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:21:50.817287 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:21:50.817288 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:21:50.817367 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:21:50.817559 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:21:50.817561 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:21:50.817562 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:21:50.817791 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:21:50.817920 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0808 22:21:50.818062 (+ 142us) server_negotiation.cc:299] Negotiation successful
0808 22:21:50.818149 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 22:21:50.824055 10762 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37240:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 22:21:50.827359 10762 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 22:21:50.842501 11479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:50.836731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.193:33471 (local address 127.0.0.1:51828)
0808 22:21:50.837772 (+ 1041us) negotiation.cc:107] Waiting for socket to connect
0808 22:21:50.837821 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0808 22:21:50.837941 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:21:50.838527 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:21:50.838533 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:21:50.838554 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:21:50.838692 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:21:50.838699 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:21:50.839793 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:21:50.839796 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:21:50.841127 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:21:50.841137 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:21:50.841967 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:21:50.841971 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:21:50.842080 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:50.842119 (+ 39us) client_negotiation.cc:769] Sending connection context
0808 22:21:50.842237 (+ 118us) client_negotiation.cc:240] Negotiation successful
0808 22:21:50.842354 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":980,"mutex_wait_us":3,"thread_start_us":94,"threads_started":1}
I20260808 22:21:50.842537 11475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:50.837088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51828 (local address 127.9.176.193:33471)
0808 22:21:50.837343 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0808 22:21:50.837359 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 22:21:50.838109 (+ 750us) server_negotiation.cc:407] Connection header received
0808 22:21:50.838193 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:21:50.838197 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:21:50.838346 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:21:50.838517 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:21:50.838836 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:50.839667 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:50.841240 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:50.841846 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:50.841953 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:50.842155 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0808 22:21:50.842294 (+ 139us) server_negotiation.cc:299] Negotiation successful
0808 22:21:50.842395 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":61,"threads_started":1}
I20260808 22:21:50.843312 11471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:50.837318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.194:43515 (local address 127.0.0.1:37178)
0808 22:21:50.838055 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0808 22:21:50.838067 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 22:21:50.838139 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:21:50.838789 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:21:50.838792 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:21:50.838811 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:21:50.838957 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:21:50.838963 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:21:50.840306 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:21:50.840309 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:21:50.841592 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:21:50.841603 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:21:50.842830 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:21:50.842833 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:21:50.842944 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:50.842992 (+ 48us) client_negotiation.cc:769] Sending connection context
0808 22:21:50.843100 (+ 108us) client_negotiation.cc:240] Negotiation successful
0808 22:21:50.843183 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":680,"mutex_wait_us":52,"thread_start_us":130,"threads_started":1}
I20260808 22:21:50.843420 11474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:50.836753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54916 (local address 127.9.176.195:38799)
0808 22:21:50.837256 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0808 22:21:50.837268 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 22:21:50.837905 (+ 637us) server_negotiation.cc:407] Connection header received
0808 22:21:50.838152 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:21:50.838159 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:21:50.838359 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:21:50.838477 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:21:50.838919 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:50.840080 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:50.841725 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:50.842532 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:50.842655 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:50.842778 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0808 22:21:50.843172 (+ 394us) server_negotiation.cc:299] Negotiation successful
0808 22:21:50.843272 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":187,"threads_started":1}
I20260808 22:21:50.843364 11480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:50.837491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37178 (local address 127.9.176.194:43515)
0808 22:21:50.837925 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0808 22:21:50.837936 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 22:21:50.838240 (+ 304us) server_negotiation.cc:407] Connection header received
0808 22:21:50.838316 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:21:50.838320 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:21:50.838475 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:21:50.838682 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:21:50.839079 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:50.840179 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:50.841753 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:50.842526 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:50.842630 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:50.842768 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0808 22:21:50.843133 (+ 365us) server_negotiation.cc:299] Negotiation successful
0808 22:21:50.843225 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":144,"threads_started":1}
I20260808 22:21:50.844213 11472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:50.836577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.195:38799 (local address 127.0.0.1:54916)
0808 22:21:50.837766 (+ 1189us) negotiation.cc:107] Waiting for socket to connect
0808 22:21:50.837816 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0808 22:21:50.837983 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:21:50.838681 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:21:50.838684 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:21:50.838701 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:21:50.838814 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:21:50.838821 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:21:50.840263 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:21:50.840266 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:21:50.841591 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:21:50.841603 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:21:50.842838 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:21:50.842841 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:21:50.842936 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:50.842987 (+ 51us) client_negotiation.cc:769] Sending connection context
0808 22:21:50.843934 (+ 947us) client_negotiation.cc:240] Negotiation successful
0808 22:21:50.844016 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1005,"mutex_wait_us":114,"thread_start_us":87,"threads_started":1}
I20260808 22:21:50.846215 10948 tablet_service.cc:1467] Processing CreateTablet for tablet 515407a38cc7422a8c8ef556a3eb65cc (DEFAULT_TABLE table=test-table [id=88f4a49b39e148b5ac294ea65e27f0aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:21:50.846272 11166 tablet_service.cc:1467] Processing CreateTablet for tablet 515407a38cc7422a8c8ef556a3eb65cc (DEFAULT_TABLE table=test-table [id=88f4a49b39e148b5ac294ea65e27f0aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:21:50.847347 10948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 515407a38cc7422a8c8ef556a3eb65cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:21:50.847036 11359 tablet_service.cc:1467] Processing CreateTablet for tablet 515407a38cc7422a8c8ef556a3eb65cc (DEFAULT_TABLE table=test-table [id=88f4a49b39e148b5ac294ea65e27f0aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:21:50.847421 11166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 515407a38cc7422a8c8ef556a3eb65cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:21:50.847957 11359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 515407a38cc7422a8c8ef556a3eb65cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:21:50.853982 11482 tablet_bootstrap.cc:492] T 515407a38cc7422a8c8ef556a3eb65cc P 19d8135fb1aa4ed184b4960acd4c4eea: Bootstrap starting.
I20260808 22:21:50.854902 11484 tablet_bootstrap.cc:492] T 515407a38cc7422a8c8ef556a3eb65cc P 27a4e31052d24e93b13a7417a967ebec: Bootstrap starting.
I20260808 22:21:50.856312 11482 tablet_bootstrap.cc:654] T 515407a38cc7422a8c8ef556a3eb65cc P 19d8135fb1aa4ed184b4960acd4c4eea: Neither blocks nor log segments found. Creating new log.
I20260808 22:21:50.857156 11484 tablet_bootstrap.cc:654] T 515407a38cc7422a8c8ef556a3eb65cc P 27a4e31052d24e93b13a7417a967ebec: Neither blocks nor log segments found. Creating new log.
I20260808 22:21:50.857240 11483 tablet_bootstrap.cc:492] T 515407a38cc7422a8c8ef556a3eb65cc P 9e3683d2459a4794b112a81f78cec799: Bootstrap starting.
I20260808 22:21:50.857721 11482 log.cc:824] T 515407a38cc7422a8c8ef556a3eb65cc P 19d8135fb1aa4ed184b4960acd4c4eea: Log is configured to *not* fsync() on all Append() calls
I20260808 22:21:50.857999 11484 log.cc:824] T 515407a38cc7422a8c8ef556a3eb65cc P 27a4e31052d24e93b13a7417a967ebec: Log is configured to *not* fsync() on all Append() calls
I20260808 22:21:50.859313 11483 tablet_bootstrap.cc:654] T 515407a38cc7422a8c8ef556a3eb65cc P 9e3683d2459a4794b112a81f78cec799: Neither blocks nor log segments found. Creating new log.
I20260808 22:21:50.859983 11482 tablet_bootstrap.cc:492] T 515407a38cc7422a8c8ef556a3eb65cc P 19d8135fb1aa4ed184b4960acd4c4eea: No bootstrap required, opened a new log
I20260808 22:21:50.859984 11484 tablet_bootstrap.cc:492] T 515407a38cc7422a8c8ef556a3eb65cc P 27a4e31052d24e93b13a7417a967ebec: No bootstrap required, opened a new log
I20260808 22:21:50.860176 11482 ts_tablet_manager.cc:1397] T 515407a38cc7422a8c8ef556a3eb65cc P 19d8135fb1aa4ed184b4960acd4c4eea: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 22:21:50.860176 11484 ts_tablet_manager.cc:1397] T 515407a38cc7422a8c8ef556a3eb65cc P 27a4e31052d24e93b13a7417a967ebec: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260808 22:21:50.860846 11483 log.cc:824] T 515407a38cc7422a8c8ef556a3eb65cc P 9e3683d2459a4794b112a81f78cec799: Log is configured to *not* fsync() on all Append() calls
I20260808 22:21:50.862466 11483 tablet_bootstrap.cc:492] T 515407a38cc7422a8c8ef556a3eb65cc P 9e3683d2459a4794b112a81f78cec799: No bootstrap required, opened a new log
I20260808 22:21:50.862637 11483 ts_tablet_manager.cc:1397] T 515407a38cc7422a8c8ef556a3eb65cc P 9e3683d2459a4794b112a81f78cec799: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 22:21:50.864130 11482 raft_consensus.cc:348] T 515407a38cc7422a8c8ef556a3eb65cc P 19d8135fb1aa4ed184b4960acd4c4eea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3683d2459a4794b112a81f78cec799" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38799 } } peers { permanent_uuid: "19d8135fb1aa4ed184b4960acd4c4eea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33471 } } peers { permanent_uuid: "27a4e31052d24e93b13a7417a967ebec" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43515 } }
I20260808 22:21:50.864122 11484 raft_consensus.cc:348] T 515407a38cc7422a8c8ef556a3eb65cc P 27a4e31052d24e93b13a7417a967ebec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3683d2459a4794b112a81f78cec799" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38799 } } peers { permanent_uuid: "19d8135fb1aa4ed184b4960acd4c4eea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33471 } } peers { permanent_uuid: "27a4e31052d24e93b13a7417a967ebec" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43515 } }
I20260808 22:21:50.864390 11484 raft_consensus.cc:374] T 515407a38cc7422a8c8ef556a3eb65cc P 27a4e31052d24e93b13a7417a967ebec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:21:50.864477 11484 raft_consensus.cc:729] T 515407a38cc7422a8c8ef556a3eb65cc P 27a4e31052d24e93b13a7417a967ebec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27a4e31052d24e93b13a7417a967ebec, State: Initialized, Role: FOLLOWER
I20260808 22:21:50.864389 11482 raft_consensus.cc:374] T 515407a38cc7422a8c8ef556a3eb65cc P 19d8135fb1aa4ed184b4960acd4c4eea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:21:50.864959 11482 raft_consensus.cc:729] T 515407a38cc7422a8c8ef556a3eb65cc P 19d8135fb1aa4ed184b4960acd4c4eea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19d8135fb1aa4ed184b4960acd4c4eea, State: Initialized, Role: FOLLOWER
I20260808 22:21:50.865029 11484 consensus_queue.cc:260] T 515407a38cc7422a8c8ef556a3eb65cc P 27a4e31052d24e93b13a7417a967ebec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3683d2459a4794b112a81f78cec799" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38799 } } peers { permanent_uuid: "19d8135fb1aa4ed184b4960acd4c4eea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33471 } } peers { permanent_uuid: "27a4e31052d24e93b13a7417a967ebec" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43515 } }
I20260808 22:21:50.865442 11482 consensus_queue.cc:260] T 515407a38cc7422a8c8ef556a3eb65cc P 19d8135fb1aa4ed184b4960acd4c4eea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3683d2459a4794b112a81f78cec799" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38799 } } peers { permanent_uuid: "19d8135fb1aa4ed184b4960acd4c4eea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33471 } } peers { permanent_uuid: "27a4e31052d24e93b13a7417a967ebec" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43515 } }
I20260808 22:21:50.865825 11258 heartbeater.cc:503] Master 127.9.176.254:44265 was elected leader, sending a full tablet report...
I20260808 22:21:50.865688 11483 raft_consensus.cc:348] T 515407a38cc7422a8c8ef556a3eb65cc P 9e3683d2459a4794b112a81f78cec799 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3683d2459a4794b112a81f78cec799" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38799 } } peers { permanent_uuid: "19d8135fb1aa4ed184b4960acd4c4eea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33471 } } peers { permanent_uuid: "27a4e31052d24e93b13a7417a967ebec" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43515 } }
I20260808 22:21:50.865921 11483 raft_consensus.cc:374] T 515407a38cc7422a8c8ef556a3eb65cc P 9e3683d2459a4794b112a81f78cec799 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:21:50.865988 11483 raft_consensus.cc:729] T 515407a38cc7422a8c8ef556a3eb65cc P 9e3683d2459a4794b112a81f78cec799 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e3683d2459a4794b112a81f78cec799, State: Initialized, Role: FOLLOWER
I20260808 22:21:50.866130 11484 ts_tablet_manager.cc:1428] T 515407a38cc7422a8c8ef556a3eb65cc P 27a4e31052d24e93b13a7417a967ebec: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260808 22:21:50.866372 11483 consensus_queue.cc:260] T 515407a38cc7422a8c8ef556a3eb65cc P 9e3683d2459a4794b112a81f78cec799 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3683d2459a4794b112a81f78cec799" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38799 } } peers { permanent_uuid: "19d8135fb1aa4ed184b4960acd4c4eea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33471 } } peers { permanent_uuid: "27a4e31052d24e93b13a7417a967ebec" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43515 } }
I20260808 22:21:50.866736 11047 heartbeater.cc:503] Master 127.9.176.254:44265 was elected leader, sending a full tablet report...
I20260808 22:21:50.866957 11482 ts_tablet_manager.cc:1428] T 515407a38cc7422a8c8ef556a3eb65cc P 19d8135fb1aa4ed184b4960acd4c4eea: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260808 22:21:50.867331 11483 ts_tablet_manager.cc:1428] T 515407a38cc7422a8c8ef556a3eb65cc P 9e3683d2459a4794b112a81f78cec799: Time spent starting tablet: real 0.005s user 0.000s sys 0.005s
I20260808 22:21:50.867672 11456 heartbeater.cc:503] Master 127.9.176.254:44265 was elected leader, sending a full tablet report...
I20260808 22:21:50.939165 11490 raft_consensus.cc:482] T 515407a38cc7422a8c8ef556a3eb65cc P 27a4e31052d24e93b13a7417a967ebec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 22:21:50.939373 11490 raft_consensus.cc:504] T 515407a38cc7422a8c8ef556a3eb65cc P 27a4e31052d24e93b13a7417a967ebec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3683d2459a4794b112a81f78cec799" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38799 } } peers { permanent_uuid: "19d8135fb1aa4ed184b4960acd4c4eea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33471 } } peers { permanent_uuid: "27a4e31052d24e93b13a7417a967ebec" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43515 } }
I20260808 22:21:50.940474 11490 leader_election.cc:288] T 515407a38cc7422a8c8ef556a3eb65cc P 27a4e31052d24e93b13a7417a967ebec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9e3683d2459a4794b112a81f78cec799 (127.9.176.195:38799), 19d8135fb1aa4ed184b4960acd4c4eea (127.9.176.193:33471)
I20260808 22:21:50.944621 11261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:50.940687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.193:33471 (local address 127.9.176.194:39875)
0808 22:21:50.940798 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0808 22:21:50.940813 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 22:21:50.940926 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:21:50.941169 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:21:50.941171 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:21:50.941188 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:21:50.941307 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:21:50.941313 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:21:50.942146 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:21:50.942149 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:21:50.943054 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:21:50.943067 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:21:50.944050 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:21:50.944054 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:21:50.944255 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:50.944299 (+ 44us) client_negotiation.cc:769] Sending connection context
0808 22:21:50.944379 (+ 80us) client_negotiation.cc:240] Negotiation successful
0808 22:21:50.944452 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":41}
I20260808 22:21:50.944667 11493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:50.940688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.195:38799 (local address 127.9.176.194:49037)
0808 22:21:50.941167 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0808 22:21:50.941191 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0808 22:21:50.941285 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:21:50.941498 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:21:50.941501 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:21:50.941517 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:21:50.941631 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:21:50.941637 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:21:50.942364 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:21:50.942367 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:21:50.943231 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:21:50.943243 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:21:50.944184 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:21:50.944187 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:21:50.944298 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:50.944324 (+ 26us) client_negotiation.cc:769] Sending connection context
0808 22:21:50.944432 (+ 108us) client_negotiation.cc:240] Negotiation successful
0808 22:21:50.944503 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"mutex_wait_us":115,"thread_start_us":144,"threads_started":1}
I20260808 22:21:50.944774 11474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:50.940798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.194:49037 (local address 127.9.176.195:38799)
0808 22:21:50.940936 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0808 22:21:50.940941 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:21:50.941276 (+ 335us) server_negotiation.cc:407] Connection header received
0808 22:21:50.941362 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:21:50.941365 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:21:50.941416 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:21:50.941474 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:21:50.941725 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:50.942257 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:50.943371 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:50.943946 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:50.944028 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:50.944080 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0808 22:21:50.944483 (+ 403us) server_negotiation.cc:299] Negotiation successful
0808 22:21:50.944559 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 22:21:50.945492 11390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "515407a38cc7422a8c8ef556a3eb65cc" candidate_uuid: "27a4e31052d24e93b13a7417a967ebec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e3683d2459a4794b112a81f78cec799" is_pre_election: true
I20260808 22:21:50.945854 11390 raft_consensus.cc:2454] T 515407a38cc7422a8c8ef556a3eb65cc P 9e3683d2459a4794b112a81f78cec799 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27a4e31052d24e93b13a7417a967ebec in term 0.
I20260808 22:21:50.946463 11094 leader_election.cc:302] T 515407a38cc7422a8c8ef556a3eb65cc P 27a4e31052d24e93b13a7417a967ebec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27a4e31052d24e93b13a7417a967ebec, 9e3683d2459a4794b112a81f78cec799; no voters:
I20260808 22:21:50.946563 11475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:50.940798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.194:39875 (local address 127.9.176.193:33471)
0808 22:21:50.940939 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0808 22:21:50.940943 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:21:50.940956 (+ 13us) server_negotiation.cc:407] Connection header received
0808 22:21:50.941012 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:21:50.941016 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:21:50.941065 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:21:50.941151 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:21:50.941420 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:50.942035 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:50.943259 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:50.943873 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:50.944028 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:50.944080 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0808 22:21:50.946370 (+ 2290us) server_negotiation.cc:299] Negotiation successful
0808 22:21:50.946454 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 22:21:50.946775 11490 raft_consensus.cc:2781] T 515407a38cc7422a8c8ef556a3eb65cc P 27a4e31052d24e93b13a7417a967ebec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 22:21:50.946879 11490 raft_consensus.cc:482] T 515407a38cc7422a8c8ef556a3eb65cc P 27a4e31052d24e93b13a7417a967ebec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 22:21:50.946931 11490 raft_consensus.cc:3037] T 515407a38cc7422a8c8ef556a3eb65cc P 27a4e31052d24e93b13a7417a967ebec [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:21:50.947105 10977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "515407a38cc7422a8c8ef556a3eb65cc" candidate_uuid: "27a4e31052d24e93b13a7417a967ebec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19d8135fb1aa4ed184b4960acd4c4eea" is_pre_election: true
I20260808 22:21:50.947377 10977 raft_consensus.cc:2454] T 515407a38cc7422a8c8ef556a3eb65cc P 19d8135fb1aa4ed184b4960acd4c4eea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27a4e31052d24e93b13a7417a967ebec in term 0.
I20260808 22:21:50.948184 11490 raft_consensus.cc:504] T 515407a38cc7422a8c8ef556a3eb65cc P 27a4e31052d24e93b13a7417a967ebec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3683d2459a4794b112a81f78cec799" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38799 } } peers { permanent_uuid: "19d8135fb1aa4ed184b4960acd4c4eea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33471 } } peers { permanent_uuid: "27a4e31052d24e93b13a7417a967ebec" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43515 } }
I20260808 22:21:50.948504 11490 leader_election.cc:288] T 515407a38cc7422a8c8ef556a3eb65cc P 27a4e31052d24e93b13a7417a967ebec [CANDIDATE]: Term 1 election: Requested vote from peers 9e3683d2459a4794b112a81f78cec799 (127.9.176.195:38799), 19d8135fb1aa4ed184b4960acd4c4eea (127.9.176.193:33471)
I20260808 22:21:50.948781 11390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "515407a38cc7422a8c8ef556a3eb65cc" candidate_uuid: "27a4e31052d24e93b13a7417a967ebec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e3683d2459a4794b112a81f78cec799"
I20260808 22:21:50.948832 10977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "515407a38cc7422a8c8ef556a3eb65cc" candidate_uuid: "27a4e31052d24e93b13a7417a967ebec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19d8135fb1aa4ed184b4960acd4c4eea"
I20260808 22:21:50.948953 11390 raft_consensus.cc:3037] T 515407a38cc7422a8c8ef556a3eb65cc P 9e3683d2459a4794b112a81f78cec799 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:21:50.948958 10977 raft_consensus.cc:3037] T 515407a38cc7422a8c8ef556a3eb65cc P 19d8135fb1aa4ed184b4960acd4c4eea [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:21:50.949913 11390 raft_consensus.cc:2454] T 515407a38cc7422a8c8ef556a3eb65cc P 9e3683d2459a4794b112a81f78cec799 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27a4e31052d24e93b13a7417a967ebec in term 1.
I20260808 22:21:50.950212 11094 leader_election.cc:302] T 515407a38cc7422a8c8ef556a3eb65cc P 27a4e31052d24e93b13a7417a967ebec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27a4e31052d24e93b13a7417a967ebec, 9e3683d2459a4794b112a81f78cec799; no voters:
I20260808 22:21:50.950429 11490 raft_consensus.cc:2781] T 515407a38cc7422a8c8ef556a3eb65cc P 27a4e31052d24e93b13a7417a967ebec [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:21:50.950484 10977 raft_consensus.cc:2454] T 515407a38cc7422a8c8ef556a3eb65cc P 19d8135fb1aa4ed184b4960acd4c4eea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27a4e31052d24e93b13a7417a967ebec in term 1.
I20260808 22:21:50.950656 11490 raft_consensus.cc:686] T 515407a38cc7422a8c8ef556a3eb65cc P 27a4e31052d24e93b13a7417a967ebec [term 1 LEADER]: Becoming Leader. State: Replica: 27a4e31052d24e93b13a7417a967ebec, State: Running, Role: LEADER
I20260808 22:21:50.950987 11490 consensus_queue.cc:237] T 515407a38cc7422a8c8ef556a3eb65cc P 27a4e31052d24e93b13a7417a967ebec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3683d2459a4794b112a81f78cec799" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38799 } } peers { permanent_uuid: "19d8135fb1aa4ed184b4960acd4c4eea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33471 } } peers { permanent_uuid: "27a4e31052d24e93b13a7417a967ebec" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43515 } }
I20260808 22:21:50.954322 10761 catalog_manager.cc:5495] T 515407a38cc7422a8c8ef556a3eb65cc P 27a4e31052d24e93b13a7417a967ebec reported cstate change: term changed from 0 to 1, leader changed from <none> to 27a4e31052d24e93b13a7417a967ebec (127.9.176.194). New cstate: current_term: 1 leader_uuid: "27a4e31052d24e93b13a7417a967ebec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3683d2459a4794b112a81f78cec799" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 38799 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19d8135fb1aa4ed184b4960acd4c4eea" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27a4e31052d24e93b13a7417a967ebec" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43515 } health_report { overall_health: HEALTHY } } }
Aug 08 22:21:50 dist-test-slave-ncp9 krb5kdc[10693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227709, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.176.193@KRBTEST.COM
I20260808 22:21:50.992862 11475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:50.984769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51842 (local address 127.9.176.193:33471)
0808 22:21:50.984933 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0808 22:21:50.984938 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:21:50.985117 (+ 179us) server_negotiation.cc:407] Connection header received
0808 22:21:50.985219 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:21:50.985221 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:21:50.985265 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:21:50.985330 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:21:50.985938 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:50.986426 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:50.987168 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:50.987369 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:50.987434 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:50.989745 (+ 2311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:21:50.989756 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:21:50.989771 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:21:50.989794 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:21:50.991507 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:21:50.991841 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:21:50.991845 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:21:50.991846 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:21:50.991966 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:21:50.992151 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:21:50.992153 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:21:50.992154 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:21:50.992381 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:21:50.992529 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0808 22:21:50.992659 (+ 130us) server_negotiation.cc:299] Negotiation successful
0808 22:21:50.992750 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
Aug 08 22:21:51 dist-test-slave-ncp9 krb5kdc[10693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227711, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 22:21:51 dist-test-slave-ncp9 krb5kdc[10693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227711, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.176.193@KRBTEST.COM
I20260808 22:21:51.019904 11475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:51.011952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51858 (local address 127.9.176.193:33471)
0808 22:21:51.012094 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0808 22:21:51.012098 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:21:51.012201 (+ 103us) server_negotiation.cc:407] Connection header received
0808 22:21:51.012323 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:21:51.012325 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:21:51.012368 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:21:51.012426 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:21:51.013061 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:51.013577 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:51.014315 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:51.014522 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:51.014589 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:51.017169 (+ 2580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:21:51.017172 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:21:51.017174 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:21:51.017197 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:21:51.018590 (+ 1393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:21:51.018946 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:21:51.018948 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:21:51.018949 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:21:51.019017 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:21:51.019205 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:21:51.019207 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:21:51.019208 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:21:51.019352 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:21:51.019444 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0808 22:21:51.019657 (+ 213us) server_negotiation.cc:299] Negotiation successful
0808 22:21:51.019748 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
Aug 08 22:21:51 dist-test-slave-ncp9 krb5kdc[10693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227711, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
W20260808 22:21:51.039007 11457 tablet.cc:2369] T 515407a38cc7422a8c8ef556a3eb65cc P 9e3683d2459a4794b112a81f78cec799: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 08 22:21:51 dist-test-slave-ncp9 krb5kdc[10693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227711, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.176.193@KRBTEST.COM
I20260808 22:21:51.045655 11475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:51.038092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51860 (local address 127.9.176.193:33471)
0808 22:21:51.038227 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0808 22:21:51.038231 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:21:51.038350 (+ 119us) server_negotiation.cc:407] Connection header received
0808 22:21:51.038483 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:21:51.038486 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:21:51.038525 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:21:51.038598 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:21:51.039380 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:51.039870 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:51.040659 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:51.040856 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:51.040979 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:51.043183 (+ 2204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:21:51.043186 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:21:51.043188 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:21:51.043210 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:21:51.044332 (+ 1122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:21:51.044691 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:21:51.044693 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:21:51.044694 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:21:51.044759 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:21:51.045011 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:21:51.045013 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:21:51.045014 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:21:51.045164 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:21:51.045259 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0808 22:21:51.045422 (+ 163us) server_negotiation.cc:299] Negotiation successful
0808 22:21:51.045507 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260808 22:21:51.047219 10914 tablet_service.cc:2908] Rejecting scan request for tablet 515407a38cc7422a8c8ef556a3eb65cc: Uninitialized: clean time has not yet been initialized
I20260808 22:21:51.049292 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 10852
I20260808 22:21:51.058646 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 11062
I20260808 22:21:51.070004 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 11266
I20260808 22:21:51.080559 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 10710
2026-08-08T22:21:51Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2389 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 22:21:51 dist-test-slave-ncp9 krb5kdc[11523](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 22:21:51 dist-test-slave-ncp9 krb5kdc[11523](info): set up 2 sockets
Aug 08 22:21:51 dist-test-slave-ncp9 krb5kdc[11523](info): commencing operation
krb5kdc: starting...
W20260808 22:21:52.202052 9923 mini_kdc.cc:121] Time spent starting KDC: real 1.095s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/krb5kdc/test-user.keytab.
Aug 08 22:21:52 dist-test-slave-ncp9 krb5kdc[11523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227712, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T22:21:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:21:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.176.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.176.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.176.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab.
Entry for principal kudu/127.9.176.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab.
I20260808 22:21:52.316996 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:33915
--webserver_interface=127.9.176.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab
--principal=kudu/127.9.176.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:43349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.176.254:33915
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:21:52.423492 11542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:21:52.423725 11542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:21:52.423763 11542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:21:52.427578 11542 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:21:52.427623 11542 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260808 22:21:52.427639 11542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:21:52.427659 11542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:21:52.427685 11542 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:21:52.427698 11542 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:21:52.432617 11542 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:43349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.176.254:33915
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab
--principal=kudu/127.9.176.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:33915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.176.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11542
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:21:52.433493 11542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:21:52.434370 11542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:21:52.442142 11552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:21:52.442174 11548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:21:52.442349 11542 server_base.cc:1034] running on GCE node
W20260808 22:21:52.442153 11549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:21:52.443015 11542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:21:52.444011 11542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:21:52.445165 11542 hybrid_clock.cc:648] HybridClock initialized: now 1786227712445152 us; error 28 us; skew 500 ppm
Aug 08 22:21:52 dist-test-slave-ncp9 krb5kdc[11523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227712, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:21:52.447701 11542 init.cc:377] Logged in from keytab as kudu/127.9.176.254@KRBTEST.COM (short username kudu)
I20260808 22:21:52.448503 11542 webserver.cc:460] Webserver started at http://127.9.176.254:33359/ using document root <none> and password file <none>
I20260808 22:21:52.449041 11542 fs_manager.cc:359] Metadata directory not provided
I20260808 22:21:52.449112 11542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:21:52.449292 11542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:21:52.451217 11542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/minicluster-data/master-0/data/instance:
uuid: "4533bca6608847efb2cf76f437edff0c"
format_stamp: "Formatted at 2026-08-08 22:21:52 on dist-test-slave-ncp9"
server_key: "8437b63f26964fd4370150f7e5f17000"
server_key_iv: "f0d95dbd9c1de83d02b86839de43d3a5"
server_key_version: "encryptionkey@0"
I20260808 22:21:52.451747 11542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/minicluster-data/master-0/wal/instance:
uuid: "4533bca6608847efb2cf76f437edff0c"
format_stamp: "Formatted at 2026-08-08 22:21:52 on dist-test-slave-ncp9"
server_key: "8437b63f26964fd4370150f7e5f17000"
server_key_iv: "f0d95dbd9c1de83d02b86839de43d3a5"
server_key_version: "encryptionkey@0"
I20260808 22:21:52.455165 11542 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260808 22:21:52.457340 11563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:21:52.458454 11542 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 22:21:52.458580 11542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/minicluster-data/master-0/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/minicluster-data/master-0/wal
uuid: "4533bca6608847efb2cf76f437edff0c"
format_stamp: "Formatted at 2026-08-08 22:21:52 on dist-test-slave-ncp9"
server_key: "8437b63f26964fd4370150f7e5f17000"
server_key_iv: "f0d95dbd9c1de83d02b86839de43d3a5"
server_key_version: "encryptionkey@0"
I20260808 22:21:52.458678 11542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:21:52.471391 11542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:21:52.473901 11542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:21:52.474058 11542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:21:52.481634 11542 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.254:33915
I20260808 22:21:52.481639 11652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.254:33915 every 8 connection(s)
I20260808 22:21:52.482519 11542 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
I20260808 22:21:52.485247 11653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:21:52.490644 11653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4533bca6608847efb2cf76f437edff0c: Bootstrap starting.
I20260808 22:21:52.492075 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 11542
I20260808 22:21:52.492202 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/minicluster-data/master-0/wal/instance
I20260808 22:21:52.492466 9923 external_mini_cluster.cc:1428] Setting key ae1d9c150cbc65fe1d2b7addcfdb5a2a
I20260808 22:21:52.493387 11653 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4533bca6608847efb2cf76f437edff0c: Neither blocks nor log segments found. Creating new log.
I20260808 22:21:52.494210 11653 log.cc:824] T 00000000000000000000000000000000 P 4533bca6608847efb2cf76f437edff0c: Log is configured to *not* fsync() on all Append() calls
I20260808 22:21:52.496340 11653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4533bca6608847efb2cf76f437edff0c: No bootstrap required, opened a new log
I20260808 22:21:52.500869 11653 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4533bca6608847efb2cf76f437edff0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4533bca6608847efb2cf76f437edff0c" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 33915 } }
I20260808 22:21:52.501156 11653 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4533bca6608847efb2cf76f437edff0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:21:52.501227 11653 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4533bca6608847efb2cf76f437edff0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4533bca6608847efb2cf76f437edff0c, State: Initialized, Role: FOLLOWER
Aug 08 22:21:52 dist-test-slave-ncp9 krb5kdc[11523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227712, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
I20260808 22:21:52.501736 11653 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4533bca6608847efb2cf76f437edff0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4533bca6608847efb2cf76f437edff0c" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 33915 } }
I20260808 22:21:52.501905 11653 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4533bca6608847efb2cf76f437edff0c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:21:52.501974 11653 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4533bca6608847efb2cf76f437edff0c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:21:52.502074 11653 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4533bca6608847efb2cf76f437edff0c [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:21:52.502890 11653 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4533bca6608847efb2cf76f437edff0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4533bca6608847efb2cf76f437edff0c" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 33915 } }
I20260808 22:21:52.503209 11653 leader_election.cc:302] T 00000000000000000000000000000000 P 4533bca6608847efb2cf76f437edff0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4533bca6608847efb2cf76f437edff0c; no voters:
I20260808 22:21:52.503492 11653 leader_election.cc:288] T 00000000000000000000000000000000 P 4533bca6608847efb2cf76f437edff0c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:21:52.503576 11659 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4533bca6608847efb2cf76f437edff0c [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:21:52.503748 11659 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4533bca6608847efb2cf76f437edff0c [term 1 LEADER]: Becoming Leader. State: Replica: 4533bca6608847efb2cf76f437edff0c, State: Running, Role: LEADER
I20260808 22:21:52.504105 11659 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4533bca6608847efb2cf76f437edff0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4533bca6608847efb2cf76f437edff0c" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 33915 } }
I20260808 22:21:52.504768 11653 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4533bca6608847efb2cf76f437edff0c [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:21:52.505859 11660 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4533bca6608847efb2cf76f437edff0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4533bca6608847efb2cf76f437edff0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4533bca6608847efb2cf76f437edff0c" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 33915 } } }
I20260808 22:21:52.505900 11661 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4533bca6608847efb2cf76f437edff0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4533bca6608847efb2cf76f437edff0c. Latest consensus state: current_term: 1 leader_uuid: "4533bca6608847efb2cf76f437edff0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4533bca6608847efb2cf76f437edff0c" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 33915 } } }
I20260808 22:21:52.506091 11661 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4533bca6608847efb2cf76f437edff0c [sys.catalog]: This master's current role is: LEADER
I20260808 22:21:52.506366 11660 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4533bca6608847efb2cf76f437edff0c [sys.catalog]: This master's current role is: LEADER
I20260808 22:21:52.506711 11670 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:21:52.507341 11657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:52.493598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60036 (local address 127.9.176.254:33915)
0808 22:21:52.494003 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0808 22:21:52.494019 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 22:21:52.494055 (+ 36us) server_negotiation.cc:407] Connection header received
0808 22:21:52.494782 (+ 727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:21:52.494809 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:21:52.495178 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:21:52.495457 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:21:52.496287 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:52.497334 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:52.498489 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:52.498953 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:52.499063 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:52.501859 (+ 2796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:21:52.501863 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:21:52.501876 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:21:52.501908 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:21:52.503605 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:21:52.504093 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:21:52.504098 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:21:52.504104 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:21:52.504253 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:21:52.504535 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:21:52.504538 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:21:52.504540 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:21:52.505023 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:21:52.505218 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0808 22:21:52.506092 (+ 874us) server_negotiation.cc:299] Negotiation successful
0808 22:21:52.506390 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":108,"threads_started":1}
I20260808 22:21:52.510942 11670 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:21:52.516501 11670 catalog_manager.cc:1347] Generated new cluster ID: 8ef63acae71b4f8da1b9ac49bffa20b1
I20260808 22:21:52.516553 11670 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:21:52.531358 11670 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:21:52.532253 11670 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:21:52.540076 11670 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4533bca6608847efb2cf76f437edff0c: Generated new TSK 0
I20260808 22:21:52.540717 11670 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.176.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.176.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.176.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.193.keytab.
Entry for principal kudu/127.9.176.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.193.keytab.
I20260808 22:21:52.565392 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.193:0
--local_ip_for_outbound_sockets=127.9.176.193
--webserver_interface=127.9.176.193
--webserver_port=0
--tserver_master_addrs=127.9.176.254:33915
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.193.keytab
--principal=kudu/127.9.176.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:43349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:21:52.673004 11688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:21:52.673239 11688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:21:52.673281 11688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:21:52.677146 11688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:21:52.677194 11688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:21:52.677238 11688 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 22:21:52.677289 11688 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.193
I20260808 22:21:52.682689 11688 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:43349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.193.keytab
--principal=kudu/127.9.176.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.176.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.176.254:33915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11688
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.193
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:21:52.683653 11688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:21:52.684643 11688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:21:52.693737 11700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:21:52.693709 11699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:21:52.693840 11688 server_base.cc:1034] running on GCE node
W20260808 22:21:52.693717 11702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:21:52.694331 11688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:21:52.694967 11688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:21:52.696142 11688 hybrid_clock.cc:648] HybridClock initialized: now 1786227712696125 us; error 27 us; skew 500 ppm
Aug 08 22:21:52 dist-test-slave-ncp9 krb5kdc[11523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227712, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:21:52.698868 11688 init.cc:377] Logged in from keytab as kudu/127.9.176.193@KRBTEST.COM (short username kudu)
I20260808 22:21:52.699633 11688 webserver.cc:460] Webserver started at http://127.9.176.193:35357/ using document root <none> and password file <none>
I20260808 22:21:52.700122 11688 fs_manager.cc:359] Metadata directory not provided
I20260808 22:21:52.700201 11688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:21:52.700384 11688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:21:52.702356 11688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/minicluster-data/ts-0/data/instance:
uuid: "bf9d58c0fe34448c8963c0ca696b3c1b"
format_stamp: "Formatted at 2026-08-08 22:21:52 on dist-test-slave-ncp9"
server_key: "ff44992fc7841c67e0cc77bda04e29df"
server_key_iv: "e7a5bb6fbaf680999a4a5b3ab4fc26f8"
server_key_version: "encryptionkey@0"
I20260808 22:21:52.702860 11688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/minicluster-data/ts-0/wal/instance:
uuid: "bf9d58c0fe34448c8963c0ca696b3c1b"
format_stamp: "Formatted at 2026-08-08 22:21:52 on dist-test-slave-ncp9"
server_key: "ff44992fc7841c67e0cc77bda04e29df"
server_key_iv: "e7a5bb6fbaf680999a4a5b3ab4fc26f8"
server_key_version: "encryptionkey@0"
I20260808 22:21:52.706468 11688 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 22:21:52.708704 11709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:21:52.709874 11688 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 22:21:52.709975 11688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/minicluster-data/ts-0/wal
uuid: "bf9d58c0fe34448c8963c0ca696b3c1b"
format_stamp: "Formatted at 2026-08-08 22:21:52 on dist-test-slave-ncp9"
server_key: "ff44992fc7841c67e0cc77bda04e29df"
server_key_iv: "e7a5bb6fbaf680999a4a5b3ab4fc26f8"
server_key_version: "encryptionkey@0"
I20260808 22:21:52.710060 11688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:21:52.748653 11688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:21:52.751922 11688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:21:52.752074 11688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:21:52.753552 11688 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:21:52.753607 11688 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:21:52.753670 11688 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:21:52.753713 11688 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 22:21:52 dist-test-slave-ncp9 krb5kdc[11523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227712, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.193@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
I20260808 22:21:52.763901 11688 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.193:44775
I20260808 22:21:52.763959 11886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.193:44775 every 8 connection(s)
I20260808 22:21:52.764849 11688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
I20260808 22:21:52.766772 11657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:52.754244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.193:45887 (local address 127.9.176.254:33915)
0808 22:21:52.754359 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0808 22:21:52.754364 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:21:52.756008 (+ 1644us) server_negotiation.cc:407] Connection header received
0808 22:21:52.757212 (+ 1204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:21:52.757216 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:21:52.757275 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:21:52.757353 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:21:52.758578 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:52.759348 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:52.760538 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:52.760779 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:52.760874 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:52.763457 (+ 2583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:21:52.763461 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:21:52.763464 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:21:52.763495 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:21:52.764862 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:21:52.765388 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:21:52.765391 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:21:52.765393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:21:52.765487 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:21:52.765860 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:21:52.765863 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:21:52.765865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:21:52.766066 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:21:52.766148 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0808 22:21:52.766549 (+ 401us) server_negotiation.cc:299] Negotiation successful
0808 22:21:52.766650 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260808 22:21:52.767532 11721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:52.755325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:33915 (local address 127.9.176.193:45887)
0808 22:21:52.755856 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0808 22:21:52.755900 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 22:21:52.756989 (+ 1089us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:21:52.757547 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:21:52.757556 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:21:52.757952 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:21:52.758427 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:21:52.758441 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:21:52.759481 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:21:52.759485 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:21:52.760417 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:21:52.760427 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:21:52.760946 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:21:52.760951 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:21:52.761114 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:52.761651 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:21:52.761666 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:21:52.763299 (+ 1633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:21:52.765010 (+ 1711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:21:52.765015 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:21:52.765025 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:21:52.765270 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:21:52.765575 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:21:52.765579 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:21:52.765581 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:21:52.765756 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:21:52.766153 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:21:52.766157 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:21:52.766367 (+ 210us) client_negotiation.cc:769] Sending connection context
0808 22:21:52.766513 (+ 146us) client_negotiation.cc:240] Negotiation successful
0808 22:21:52.766758 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":121,"threads_started":1}
I20260808 22:21:52.768747 11888 heartbeater.cc:347] Connected to a master server at 127.9.176.254:33915
I20260808 22:21:52.768884 11888 heartbeater.cc:464] Registering TS with master...
I20260808 22:21:52.769375 11888 heartbeater.cc:511] Master 127.9.176.254:33915 requested a full tablet report, sending...
I20260808 22:21:52.770871 11589 ts_manager.cc:194] Registered new tserver with Master: bf9d58c0fe34448c8963c0ca696b3c1b (127.9.176.193:44775)
I20260808 22:21:52.771227 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 11688
I20260808 22:21:52.771313 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/minicluster-data/ts-0/wal/instance
I20260808 22:21:52.771543 9923 external_mini_cluster.cc:1428] Setting key d56eb305edae364dcae65d978a6403f5
I20260808 22:21:52.772106 11589 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.176.193@KRBTEST.COM'} at 127.9.176.193:45887
I20260808 22:21:52.779059 11898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:52.770713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:33915 (local address 127.9.176.193:46123)
0808 22:21:52.772767 (+ 2054us) negotiation.cc:107] Waiting for socket to connect
0808 22:21:52.772781 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 22:21:52.772936 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:21:52.773189 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:21:52.773192 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:21:52.773386 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:21:52.773510 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:21:52.773517 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:21:52.774410 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:21:52.774412 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:21:52.774940 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:21:52.774948 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:21:52.775361 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:21:52.775365 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:21:52.775453 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:52.775890 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:21:52.775900 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:21:52.776183 (+ 283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:21:52.777750 (+ 1567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:21:52.777752 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:21:52.777753 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:21:52.777936 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:21:52.778232 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:21:52.778235 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:21:52.778236 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:21:52.778310 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:21:52.778683 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:21:52.778685 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:21:52.778759 (+ 74us) client_negotiation.cc:769] Sending connection context
0808 22:21:52.778833 (+ 74us) client_negotiation.cc:240] Negotiation successful
0808 22:21:52.778932 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1968,"thread_start_us":97,"threads_started":1}
I20260808 22:21:52.779170 11657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:52.770787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.193:46123 (local address 127.9.176.254:33915)
0808 22:21:52.770883 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0808 22:21:52.770886 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:21:52.772845 (+ 1959us) server_negotiation.cc:407] Connection header received
0808 22:21:52.773032 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:21:52.773035 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:21:52.773078 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:21:52.773158 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:21:52.773633 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:52.774286 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:52.775054 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:52.775264 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:52.775348 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:52.776311 (+ 963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:21:52.776315 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:21:52.776318 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:21:52.776342 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:21:52.777645 (+ 1303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:21:52.778040 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:21:52.778044 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:21:52.778046 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:21:52.778136 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:21:52.778386 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:21:52.778389 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:21:52.778391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:21:52.778591 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:21:52.778702 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0808 22:21:52.778912 (+ 210us) server_negotiation.cc:299] Negotiation successful
0808 22:21:52.779024 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
WARNING: no policy specified for kudu/127.9.176.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.176.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.176.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.194.keytab.
Entry for principal kudu/127.9.176.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.194.keytab.
I20260808 22:21:52.797515 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.194:0
--local_ip_for_outbound_sockets=127.9.176.194
--webserver_interface=127.9.176.194
--webserver_port=0
--tserver_master_addrs=127.9.176.254:33915
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.194.keytab
--principal=kudu/127.9.176.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:43349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:21:52.903090 11903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:21:52.903326 11903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:21:52.903380 11903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:21:52.907418 11903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:21:52.907471 11903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:21:52.907521 11903 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 22:21:52.907573 11903 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.194
I20260808 22:21:52.912587 11903 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:43349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.194.keytab
--principal=kudu/127.9.176.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.176.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.176.254:33915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11903
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.194
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:21:52.913537 11903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:21:52.914467 11903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:21:52.923467 11913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:21:52.923480 11910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:21:52.923477 11917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:21:52.923684 11903 server_base.cc:1034] running on GCE node
I20260808 22:21:52.924050 11903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:21:52.924639 11903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:21:52.925804 11903 hybrid_clock.cc:648] HybridClock initialized: now 1786227712925781 us; error 33 us; skew 500 ppm
Aug 08 22:21:52 dist-test-slave-ncp9 krb5kdc[11523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227712, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:21:52.928357 11903 init.cc:377] Logged in from keytab as kudu/127.9.176.194@KRBTEST.COM (short username kudu)
I20260808 22:21:52.929203 11903 webserver.cc:460] Webserver started at http://127.9.176.194:38697/ using document root <none> and password file <none>
I20260808 22:21:52.929695 11903 fs_manager.cc:359] Metadata directory not provided
I20260808 22:21:52.929761 11903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:21:52.929966 11903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:21:52.931967 11903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/minicluster-data/ts-1/data/instance:
uuid: "50ce356aabd9458197537fddca193355"
format_stamp: "Formatted at 2026-08-08 22:21:52 on dist-test-slave-ncp9"
server_key: "7c6b8d3877e03a2134d6c46177c7df46"
server_key_iv: "affa4894cfd76ea216e0c9e2505728ff"
server_key_version: "encryptionkey@0"
I20260808 22:21:52.932485 11903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/minicluster-data/ts-1/wal/instance:
uuid: "50ce356aabd9458197537fddca193355"
format_stamp: "Formatted at 2026-08-08 22:21:52 on dist-test-slave-ncp9"
server_key: "7c6b8d3877e03a2134d6c46177c7df46"
server_key_iv: "affa4894cfd76ea216e0c9e2505728ff"
server_key_version: "encryptionkey@0"
I20260808 22:21:52.936278 11903 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 22:21:52.938555 11929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:21:52.939662 11903 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 22:21:52.939780 11903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/minicluster-data/ts-1/wal
uuid: "50ce356aabd9458197537fddca193355"
format_stamp: "Formatted at 2026-08-08 22:21:52 on dist-test-slave-ncp9"
server_key: "7c6b8d3877e03a2134d6c46177c7df46"
server_key_iv: "affa4894cfd76ea216e0c9e2505728ff"
server_key_version: "encryptionkey@0"
I20260808 22:21:52.939867 11903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:21:52.960335 11903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:21:52.963285 11903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:21:52.963465 11903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:21:52.964820 11903 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:21:52.964867 11903 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:21:52.964943 11903 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:21:52.964974 11903 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 22:21:52 dist-test-slave-ncp9 krb5kdc[11523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227712, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.194@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
I20260808 22:21:52.974915 11903 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.194:46851
I20260808 22:21:52.974967 12094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.194:46851 every 8 connection(s)
I20260808 22:21:52.975770 11903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
I20260808 22:21:52.978232 11657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:52.965581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.194:41615 (local address 127.9.176.254:33915)
0808 22:21:52.965700 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0808 22:21:52.965704 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:21:52.966716 (+ 1012us) server_negotiation.cc:407] Connection header received
0808 22:21:52.967721 (+ 1005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:21:52.967725 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:21:52.967788 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:21:52.967878 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:21:52.969248 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:52.970112 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:52.971313 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:52.971649 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:52.971765 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:52.975033 (+ 3268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:21:52.975037 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:21:52.975040 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:21:52.975071 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:21:52.976566 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:21:52.977027 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:21:52.977030 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:21:52.977031 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:21:52.977113 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:21:52.977374 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:21:52.977376 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:21:52.977377 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:21:52.977538 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:21:52.977612 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0808 22:21:52.978020 (+ 408us) server_negotiation.cc:299] Negotiation successful
0808 22:21:52.978116 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 22:21:52.978911 11940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:52.966111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:33915 (local address 127.9.176.194:41615)
0808 22:21:52.966569 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0808 22:21:52.966609 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 22:21:52.967467 (+ 858us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:21:52.968077 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:21:52.968087 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:21:52.968522 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:21:52.969099 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:21:52.969115 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:21:52.970241 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:21:52.970245 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:21:52.971191 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:21:52.971201 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:21:52.971811 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:21:52.971815 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:21:52.972018 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:52.973186 (+ 1168us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:21:52.973201 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:21:52.974862 (+ 1661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:21:52.976664 (+ 1802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:21:52.976669 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:21:52.976679 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:21:52.976943 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:21:52.977181 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:21:52.977183 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:21:52.977184 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:21:52.977306 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:21:52.977609 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:21:52.977613 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:21:52.977824 (+ 211us) client_negotiation.cc:769] Sending connection context
0808 22:21:52.977963 (+ 139us) client_negotiation.cc:240] Negotiation successful
0808 22:21:52.978187 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":132,"threads_started":1}
I20260808 22:21:52.979959 12095 heartbeater.cc:347] Connected to a master server at 127.9.176.254:33915
I20260808 22:21:52.980085 12095 heartbeater.cc:464] Registering TS with master...
I20260808 22:21:52.980521 12095 heartbeater.cc:511] Master 127.9.176.254:33915 requested a full tablet report, sending...
I20260808 22:21:52.981580 11590 ts_manager.cc:194] Registered new tserver with Master: 50ce356aabd9458197537fddca193355 (127.9.176.194:46851)
I20260808 22:21:52.982291 11590 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.176.194@KRBTEST.COM'} at 127.9.176.194:41615
I20260808 22:21:52.982483 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 11903
I20260808 22:21:52.982581 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/minicluster-data/ts-1/wal/instance
I20260808 22:21:52.982889 9923 external_mini_cluster.cc:1428] Setting key 5641a7125dca100b1efcee4b5dedf56c
I20260808 22:21:52.988847 11657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:52.981965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.194:46153 (local address 127.9.176.254:33915)
0808 22:21:52.982065 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0808 22:21:52.982069 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:21:52.982256 (+ 187us) server_negotiation.cc:407] Connection header received
0808 22:21:52.982349 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:21:52.982352 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:21:52.982391 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:21:52.982455 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:21:52.982888 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:52.983598 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:52.984641 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:52.984982 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:52.985080 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:52.986065 (+ 985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:21:52.986067 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:21:52.986069 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:21:52.986089 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:21:52.987297 (+ 1208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:21:52.987746 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:21:52.987749 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:21:52.987750 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:21:52.987830 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:21:52.988086 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:21:52.988088 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:21:52.988089 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:21:52.988270 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:21:52.988344 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0808 22:21:52.988639 (+ 295us) server_negotiation.cc:299] Negotiation successful
0808 22:21:52.988744 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260808 22:21:52.988921 12102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:52.981906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:33915 (local address 127.9.176.194:46153)
0808 22:21:52.982173 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0808 22:21:52.982194 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 22:21:52.982280 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:21:52.982480 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:21:52.982482 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:21:52.982686 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:21:52.982781 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:21:52.982787 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:21:52.983758 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:21:52.983761 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:21:52.984520 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:21:52.984529 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:21:52.985077 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:21:52.985079 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:21:52.985148 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:52.985555 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:21:52.985566 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:21:52.985937 (+ 371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:21:52.987402 (+ 1465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:21:52.987406 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:21:52.987407 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:21:52.987630 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:21:52.987919 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:21:52.987921 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:21:52.987923 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:21:52.988012 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:21:52.988368 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:21:52.988371 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:21:52.988469 (+ 98us) client_negotiation.cc:769] Sending connection context
0808 22:21:52.988575 (+ 106us) client_negotiation.cc:240] Negotiation successful
0808 22:21:52.988712 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":111,"threads_started":1}
WARNING: no policy specified for kudu/127.9.176.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.176.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.176.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.195.keytab.
Entry for principal kudu/127.9.176.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.195.keytab.
I20260808 22:21:53.007814 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.195:0
--local_ip_for_outbound_sockets=127.9.176.195
--webserver_interface=127.9.176.195
--webserver_port=0
--tserver_master_addrs=127.9.176.254:33915
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.195.keytab
--principal=kudu/127.9.176.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:43349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:21:53.115950 12106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:21:53.116195 12106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:21:53.116253 12106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:21:53.120292 12106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:21:53.120347 12106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:21:53.120397 12106 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 22:21:53.120448 12106 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.195
I20260808 22:21:53.125478 12106 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:43349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.195.keytab
--principal=kudu/127.9.176.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.176.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.176.254:33915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12106
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.195
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:21:53.126389 12106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:21:53.127311 12106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:21:53.136274 12112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:21:53.136284 12113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:21:53.136381 12106 server_base.cc:1034] running on GCE node
W20260808 22:21:53.136276 12115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:21:53.136957 12106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:21:53.137533 12106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:21:53.138688 12106 hybrid_clock.cc:648] HybridClock initialized: now 1786227713138663 us; error 39 us; skew 500 ppm
Aug 08 22:21:53 dist-test-slave-ncp9 krb5kdc[11523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227713, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:21:53.141310 12106 init.cc:377] Logged in from keytab as kudu/127.9.176.195@KRBTEST.COM (short username kudu)
I20260808 22:21:53.142225 12106 webserver.cc:460] Webserver started at http://127.9.176.195:39841/ using document root <none> and password file <none>
I20260808 22:21:53.142757 12106 fs_manager.cc:359] Metadata directory not provided
I20260808 22:21:53.142824 12106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:21:53.143008 12106 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:21:53.145076 12106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/minicluster-data/ts-2/data/instance:
uuid: "1fd4acc8159e428bb11f17af176c3542"
format_stamp: "Formatted at 2026-08-08 22:21:53 on dist-test-slave-ncp9"
server_key: "e8d19544acb697dca78bb3118b739a58"
server_key_iv: "7739b9b55f716ce2a273c18a3641a1af"
server_key_version: "encryptionkey@0"
I20260808 22:21:53.145607 12106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/minicluster-data/ts-2/wal/instance:
uuid: "1fd4acc8159e428bb11f17af176c3542"
format_stamp: "Formatted at 2026-08-08 22:21:53 on dist-test-slave-ncp9"
server_key: "e8d19544acb697dca78bb3118b739a58"
server_key_iv: "7739b9b55f716ce2a273c18a3641a1af"
server_key_version: "encryptionkey@0"
I20260808 22:21:53.149101 12106 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260808 22:21:53.151365 12124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:21:53.152503 12106 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 22:21:53.152608 12106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/minicluster-data/ts-2/wal
uuid: "1fd4acc8159e428bb11f17af176c3542"
format_stamp: "Formatted at 2026-08-08 22:21:53 on dist-test-slave-ncp9"
server_key: "e8d19544acb697dca78bb3118b739a58"
server_key_iv: "7739b9b55f716ce2a273c18a3641a1af"
server_key_version: "encryptionkey@0"
I20260808 22:21:53.152705 12106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:21:53.169415 12106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:21:53.172315 12106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:21:53.172482 12106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:21:53.173909 12106 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:21:53.173969 12106 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:21:53.174022 12106 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:21:53.174052 12106 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 22:21:53 dist-test-slave-ncp9 krb5kdc[11523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227713, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.195@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
I20260808 22:21:53.184324 12106 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.195:46643
I20260808 22:21:53.184623 12286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.195:46643 every 8 connection(s)
I20260808 22:21:53.185317 12106 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
I20260808 22:21:53.187117 11657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:53.174569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.195:60813 (local address 127.9.176.254:33915)
0808 22:21:53.174698 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0808 22:21:53.174703 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:21:53.176233 (+ 1530us) server_negotiation.cc:407] Connection header received
0808 22:21:53.177387 (+ 1154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:21:53.177391 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:21:53.177451 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:21:53.177543 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:21:53.178695 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:53.179326 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:53.180638 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:53.181019 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:53.181158 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:53.183585 (+ 2427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:21:53.183589 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:21:53.183591 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:21:53.183621 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:21:53.185133 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:21:53.185641 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:21:53.185644 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:21:53.185646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:21:53.185741 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:21:53.186094 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:21:53.186097 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:21:53.186099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:21:53.186279 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:21:53.186366 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0808 22:21:53.186834 (+ 468us) server_negotiation.cc:299] Negotiation successful
0808 22:21:53.186959 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260808 22:21:53.187814 12137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:53.175601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:33915 (local address 127.9.176.195:60813)
0808 22:21:53.176094 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0808 22:21:53.176136 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 22:21:53.177150 (+ 1014us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:21:53.177687 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:21:53.177693 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:21:53.178085 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:21:53.178534 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:21:53.178547 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:21:53.179440 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:21:53.179444 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:21:53.180520 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:21:53.180530 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:21:53.181192 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:21:53.181196 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:21:53.181365 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:53.181864 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:21:53.181879 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:21:53.183429 (+ 1550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:21:53.185244 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:21:53.185250 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:21:53.185264 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:21:53.185547 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:21:53.185849 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:21:53.185852 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:21:53.185853 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:21:53.186010 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:21:53.186379 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:21:53.186386 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:21:53.186609 (+ 223us) client_negotiation.cc:769] Sending connection context
0808 22:21:53.186770 (+ 161us) client_negotiation.cc:240] Negotiation successful
0808 22:21:53.187063 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":135,"threads_started":1}
I20260808 22:21:53.188948 12287 heartbeater.cc:347] Connected to a master server at 127.9.176.254:33915
I20260808 22:21:53.189090 12287 heartbeater.cc:464] Registering TS with master...
I20260808 22:21:53.189575 12287 heartbeater.cc:511] Master 127.9.176.254:33915 requested a full tablet report, sending...
I20260808 22:21:53.190603 11589 ts_manager.cc:194] Registered new tserver with Master: 1fd4acc8159e428bb11f17af176c3542 (127.9.176.195:46643)
I20260808 22:21:53.191207 11589 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.176.195@KRBTEST.COM'} at 127.9.176.195:60813
I20260808 22:21:53.193109 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 12106
I20260808 22:21:53.193228 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/minicluster-data/ts-2/wal/instance
I20260808 22:21:53.193477 9923 external_mini_cluster.cc:1428] Setting key c2fbbf6e869cbdf68da1993ba159b072
I20260808 22:21:53.195463 9923 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 22:21:53.197345 11657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:53.190917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.195:34697 (local address 127.9.176.254:33915)
0808 22:21:53.191037 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0808 22:21:53.191041 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:21:53.191206 (+ 165us) server_negotiation.cc:407] Connection header received
0808 22:21:53.191297 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:21:53.191300 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:21:53.191338 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:21:53.191398 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:21:53.191816 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:53.192303 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:53.193361 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:53.193579 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:53.193667 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:53.194642 (+ 975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:21:53.194646 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:21:53.194648 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:21:53.194672 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:21:53.195892 (+ 1220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:21:53.196263 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:21:53.196266 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:21:53.196268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:21:53.196344 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:21:53.196579 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:21:53.196582 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:21:53.196583 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:21:53.196750 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:21:53.196832 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0808 22:21:53.197118 (+ 286us) server_negotiation.cc:299] Negotiation successful
0808 22:21:53.197215 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 22:21:53.197355 12293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:53.190843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:33915 (local address 127.9.176.195:34697)
0808 22:21:53.191131 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0808 22:21:53.191142 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 22:21:53.191228 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:21:53.191423 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:21:53.191425 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:21:53.191596 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:21:53.191699 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:21:53.191706 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:21:53.192426 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:21:53.192429 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:21:53.193224 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:21:53.193235 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:21:53.193681 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:21:53.193685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:21:53.193777 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:53.194193 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:21:53.194206 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:21:53.194534 (+ 328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:21:53.195990 (+ 1456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:21:53.195993 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:21:53.195995 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:21:53.196177 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:21:53.196423 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:21:53.196426 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:21:53.196428 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:21:53.196500 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:21:53.196839 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:21:53.196842 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:21:53.196952 (+ 110us) client_negotiation.cc:769] Sending connection context
0808 22:21:53.197043 (+ 91us) client_negotiation.cc:240] Negotiation successful
0808 22:21:53.197184 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":80,"threads_started":1}
Aug 08 22:21:53 dist-test-slave-ncp9 krb5kdc[11523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227713, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 22:21:53 dist-test-slave-ncp9 krb5kdc[11523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227713, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
I20260808 22:21:53.225802 11657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:53.214881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60042 (local address 127.9.176.254:33915)
0808 22:21:53.215005 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0808 22:21:53.215009 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:21:53.215020 (+ 11us) server_negotiation.cc:407] Connection header received
0808 22:21:53.215059 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:21:53.215062 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:21:53.215098 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:21:53.215165 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:21:53.215860 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:53.216402 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:53.217326 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:53.217545 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:53.217635 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:53.223186 (+ 5551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:21:53.223189 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:21:53.223192 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:21:53.223218 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:21:53.224478 (+ 1260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:21:53.224795 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:21:53.224798 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:21:53.224799 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:21:53.224870 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:21:53.225089 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:21:53.225091 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:21:53.225092 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:21:53.225276 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:21:53.225379 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0808 22:21:53.225546 (+ 167us) server_negotiation.cc:299] Negotiation successful
0808 22:21:53.225641 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260808 22:21:53.228654 11589 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60042:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 22:21:53.231652 11589 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 22:21:53.245569 12309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:53.239539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.195:46643 (local address 127.0.0.1:58068)
0808 22:21:53.240341 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0808 22:21:53.240392 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0808 22:21:53.240512 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:21:53.241471 (+ 959us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:21:53.241477 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:21:53.241498 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:21:53.241633 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:21:53.241640 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:21:53.242784 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:21:53.242787 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:21:53.244101 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:21:53.244112 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:21:53.245031 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:21:53.245035 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:21:53.245150 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:53.245194 (+ 44us) client_negotiation.cc:769] Sending connection context
0808 22:21:53.245309 (+ 115us) client_negotiation.cc:240] Negotiation successful
0808 22:21:53.245426 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"thread_start_us":89,"threads_started":1}
I20260808 22:21:53.245592 12312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:53.239687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58068 (local address 127.9.176.195:46643)
0808 22:21:53.241054 (+ 1367us) server_negotiation.cc:206] Beginning negotiation
0808 22:21:53.241065 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 22:21:53.241087 (+ 22us) server_negotiation.cc:407] Connection header received
0808 22:21:53.241176 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:21:53.241181 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:21:53.241329 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:21:53.241466 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:21:53.241780 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:53.242637 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:53.244219 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:53.244888 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:53.245030 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:53.245197 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0808 22:21:53.245352 (+ 155us) server_negotiation.cc:299] Negotiation successful
0808 22:21:53.245451 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1267,"thread_start_us":115,"threads_started":1}
I20260808 22:21:53.246016 12313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:53.239643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.194:46851 (local address 127.0.0.1:40638)
0808 22:21:53.240775 (+ 1132us) negotiation.cc:107] Waiting for socket to connect
0808 22:21:53.240788 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 22:21:53.240886 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:21:53.241629 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:21:53.241632 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:21:53.241647 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:21:53.241720 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:21:53.241725 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:21:53.243196 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:21:53.243199 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:21:53.244531 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:21:53.244540 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:21:53.245639 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:21:53.245643 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:21:53.245740 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:53.245765 (+ 25us) client_negotiation.cc:769] Sending connection context
0808 22:21:53.245834 (+ 69us) client_negotiation.cc:240] Negotiation successful
0808 22:21:53.245892 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1068,"mutex_wait_us":148,"thread_start_us":101,"threads_started":1}
I20260808 22:21:53.246357 12314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:53.240369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40638 (local address 127.9.176.194:46851)
0808 22:21:53.240792 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0808 22:21:53.240804 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 22:21:53.241009 (+ 205us) server_negotiation.cc:407] Connection header received
0808 22:21:53.241096 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:21:53.241101 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:21:53.241311 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:21:53.241472 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:21:53.241821 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:53.243063 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:53.244620 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:53.245339 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:53.245428 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:53.245573 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0808 22:21:53.245871 (+ 298us) server_negotiation.cc:299] Negotiation successful
0808 22:21:53.245967 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":97,"threads_started":1}
I20260808 22:21:53.248595 12198 tablet_service.cc:1467] Processing CreateTablet for tablet 68719037a0af4cbdbe562b557293b9df (DEFAULT_TABLE table=test-table [id=0bf2c5b3e4bf4ec9ad583f6fb2181932]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:21:53.248600 12002 tablet_service.cc:1467] Processing CreateTablet for tablet 68719037a0af4cbdbe562b557293b9df (DEFAULT_TABLE table=test-table [id=0bf2c5b3e4bf4ec9ad583f6fb2181932]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:21:53.249264 12317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:53.240490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36790 (local address 127.9.176.193:44775)
0808 22:21:53.240826 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0808 22:21:53.240840 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 22:21:53.241070 (+ 230us) server_negotiation.cc:407] Connection header received
0808 22:21:53.241219 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:21:53.241223 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:21:53.241372 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:21:53.241490 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:21:53.241953 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:53.243100 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:53.244550 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:53.245387 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:53.245498 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:53.245650 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0808 22:21:53.249011 (+ 3361us) server_negotiation.cc:299] Negotiation successful
0808 22:21:53.249110 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":74,"threads_started":1}
I20260808 22:21:53.249626 12311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:53.239925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.193:44775 (local address 127.0.0.1:36790)
0808 22:21:53.240960 (+ 1035us) negotiation.cc:107] Waiting for socket to connect
0808 22:21:53.240973 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 22:21:53.241076 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:21:53.241654 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:21:53.241664 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:21:53.241686 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:21:53.241799 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:21:53.241806 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:21:53.243218 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:21:53.243221 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:21:53.244425 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:21:53.244437 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:21:53.245718 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:21:53.245722 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:21:53.245813 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:53.245837 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 22:21:53.245906 (+ 69us) client_negotiation.cc:240] Negotiation successful
0808 22:21:53.249503 (+ 3597us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":958,"mutex_wait_us":36,"thread_start_us":90,"threads_started":1}
I20260808 22:21:53.249641 12198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68719037a0af4cbdbe562b557293b9df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:21:53.249645 12002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68719037a0af4cbdbe562b557293b9df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:21:53.252105 11785 tablet_service.cc:1467] Processing CreateTablet for tablet 68719037a0af4cbdbe562b557293b9df (DEFAULT_TABLE table=test-table [id=0bf2c5b3e4bf4ec9ad583f6fb2181932]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:21:53.253108 11785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68719037a0af4cbdbe562b557293b9df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:21:53.256362 12319 tablet_bootstrap.cc:492] T 68719037a0af4cbdbe562b557293b9df P 1fd4acc8159e428bb11f17af176c3542: Bootstrap starting.
I20260808 22:21:53.258387 12319 tablet_bootstrap.cc:654] T 68719037a0af4cbdbe562b557293b9df P 1fd4acc8159e428bb11f17af176c3542: Neither blocks nor log segments found. Creating new log.
I20260808 22:21:53.259063 12319 log.cc:824] T 68719037a0af4cbdbe562b557293b9df P 1fd4acc8159e428bb11f17af176c3542: Log is configured to *not* fsync() on all Append() calls
I20260808 22:21:53.259846 12322 tablet_bootstrap.cc:492] T 68719037a0af4cbdbe562b557293b9df P bf9d58c0fe34448c8963c0ca696b3c1b: Bootstrap starting.
I20260808 22:21:53.260725 12320 tablet_bootstrap.cc:492] T 68719037a0af4cbdbe562b557293b9df P 50ce356aabd9458197537fddca193355: Bootstrap starting.
I20260808 22:21:53.261139 12319 tablet_bootstrap.cc:492] T 68719037a0af4cbdbe562b557293b9df P 1fd4acc8159e428bb11f17af176c3542: No bootstrap required, opened a new log
I20260808 22:21:53.261324 12319 ts_tablet_manager.cc:1397] T 68719037a0af4cbdbe562b557293b9df P 1fd4acc8159e428bb11f17af176c3542: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260808 22:21:53.261982 12322 tablet_bootstrap.cc:654] T 68719037a0af4cbdbe562b557293b9df P bf9d58c0fe34448c8963c0ca696b3c1b: Neither blocks nor log segments found. Creating new log.
I20260808 22:21:53.262578 12320 tablet_bootstrap.cc:654] T 68719037a0af4cbdbe562b557293b9df P 50ce356aabd9458197537fddca193355: Neither blocks nor log segments found. Creating new log.
I20260808 22:21:53.262820 12322 log.cc:824] T 68719037a0af4cbdbe562b557293b9df P bf9d58c0fe34448c8963c0ca696b3c1b: Log is configured to *not* fsync() on all Append() calls
I20260808 22:21:53.263314 12320 log.cc:824] T 68719037a0af4cbdbe562b557293b9df P 50ce356aabd9458197537fddca193355: Log is configured to *not* fsync() on all Append() calls
I20260808 22:21:53.264767 12322 tablet_bootstrap.cc:492] T 68719037a0af4cbdbe562b557293b9df P bf9d58c0fe34448c8963c0ca696b3c1b: No bootstrap required, opened a new log
I20260808 22:21:53.264874 12320 tablet_bootstrap.cc:492] T 68719037a0af4cbdbe562b557293b9df P 50ce356aabd9458197537fddca193355: No bootstrap required, opened a new log
I20260808 22:21:53.264987 12322 ts_tablet_manager.cc:1397] T 68719037a0af4cbdbe562b557293b9df P bf9d58c0fe34448c8963c0ca696b3c1b: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260808 22:21:53.265076 12320 ts_tablet_manager.cc:1397] T 68719037a0af4cbdbe562b557293b9df P 50ce356aabd9458197537fddca193355: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260808 22:21:53.265288 12319 raft_consensus.cc:348] T 68719037a0af4cbdbe562b557293b9df P 1fd4acc8159e428bb11f17af176c3542 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fd4acc8159e428bb11f17af176c3542" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46643 } } peers { permanent_uuid: "50ce356aabd9458197537fddca193355" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 46851 } } peers { permanent_uuid: "bf9d58c0fe34448c8963c0ca696b3c1b" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44775 } }
I20260808 22:21:53.265553 12319 raft_consensus.cc:374] T 68719037a0af4cbdbe562b557293b9df P 1fd4acc8159e428bb11f17af176c3542 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:21:53.265650 12319 raft_consensus.cc:729] T 68719037a0af4cbdbe562b557293b9df P 1fd4acc8159e428bb11f17af176c3542 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fd4acc8159e428bb11f17af176c3542, State: Initialized, Role: FOLLOWER
I20260808 22:21:53.266119 12319 consensus_queue.cc:260] T 68719037a0af4cbdbe562b557293b9df P 1fd4acc8159e428bb11f17af176c3542 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fd4acc8159e428bb11f17af176c3542" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46643 } } peers { permanent_uuid: "50ce356aabd9458197537fddca193355" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 46851 } } peers { permanent_uuid: "bf9d58c0fe34448c8963c0ca696b3c1b" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44775 } }
I20260808 22:21:53.266865 12287 heartbeater.cc:503] Master 127.9.176.254:33915 was elected leader, sending a full tablet report...
I20260808 22:21:53.267199 12319 ts_tablet_manager.cc:1428] T 68719037a0af4cbdbe562b557293b9df P 1fd4acc8159e428bb11f17af176c3542: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260808 22:21:53.268795 12322 raft_consensus.cc:348] T 68719037a0af4cbdbe562b557293b9df P bf9d58c0fe34448c8963c0ca696b3c1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fd4acc8159e428bb11f17af176c3542" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46643 } } peers { permanent_uuid: "50ce356aabd9458197537fddca193355" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 46851 } } peers { permanent_uuid: "bf9d58c0fe34448c8963c0ca696b3c1b" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44775 } }
I20260808 22:21:53.269035 12322 raft_consensus.cc:374] T 68719037a0af4cbdbe562b557293b9df P bf9d58c0fe34448c8963c0ca696b3c1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:21:53.269120 12322 raft_consensus.cc:729] T 68719037a0af4cbdbe562b557293b9df P bf9d58c0fe34448c8963c0ca696b3c1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9d58c0fe34448c8963c0ca696b3c1b, State: Initialized, Role: FOLLOWER
I20260808 22:21:53.269541 12322 consensus_queue.cc:260] T 68719037a0af4cbdbe562b557293b9df P bf9d58c0fe34448c8963c0ca696b3c1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fd4acc8159e428bb11f17af176c3542" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46643 } } peers { permanent_uuid: "50ce356aabd9458197537fddca193355" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 46851 } } peers { permanent_uuid: "bf9d58c0fe34448c8963c0ca696b3c1b" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44775 } }
I20260808 22:21:53.270252 11888 heartbeater.cc:503] Master 127.9.176.254:33915 was elected leader, sending a full tablet report...
I20260808 22:21:53.270399 12322 ts_tablet_manager.cc:1428] T 68719037a0af4cbdbe562b557293b9df P bf9d58c0fe34448c8963c0ca696b3c1b: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260808 22:21:53.271039 12320 raft_consensus.cc:348] T 68719037a0af4cbdbe562b557293b9df P 50ce356aabd9458197537fddca193355 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fd4acc8159e428bb11f17af176c3542" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46643 } } peers { permanent_uuid: "50ce356aabd9458197537fddca193355" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 46851 } } peers { permanent_uuid: "bf9d58c0fe34448c8963c0ca696b3c1b" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44775 } }
I20260808 22:21:53.271309 12320 raft_consensus.cc:374] T 68719037a0af4cbdbe562b557293b9df P 50ce356aabd9458197537fddca193355 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:21:53.271384 12320 raft_consensus.cc:729] T 68719037a0af4cbdbe562b557293b9df P 50ce356aabd9458197537fddca193355 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50ce356aabd9458197537fddca193355, State: Initialized, Role: FOLLOWER
I20260808 22:21:53.271783 12320 consensus_queue.cc:260] T 68719037a0af4cbdbe562b557293b9df P 50ce356aabd9458197537fddca193355 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fd4acc8159e428bb11f17af176c3542" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46643 } } peers { permanent_uuid: "50ce356aabd9458197537fddca193355" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 46851 } } peers { permanent_uuid: "bf9d58c0fe34448c8963c0ca696b3c1b" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44775 } }
I20260808 22:21:53.272456 12095 heartbeater.cc:503] Master 127.9.176.254:33915 was elected leader, sending a full tablet report...
I20260808 22:21:53.272600 12320 ts_tablet_manager.cc:1428] T 68719037a0af4cbdbe562b557293b9df P 50ce356aabd9458197537fddca193355: Time spent starting tablet: real 0.007s user 0.000s sys 0.005s
I20260808 22:21:53.279241 12325 raft_consensus.cc:482] T 68719037a0af4cbdbe562b557293b9df P 1fd4acc8159e428bb11f17af176c3542 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 22:21:53.279385 12325 raft_consensus.cc:504] T 68719037a0af4cbdbe562b557293b9df P 1fd4acc8159e428bb11f17af176c3542 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fd4acc8159e428bb11f17af176c3542" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46643 } } peers { permanent_uuid: "50ce356aabd9458197537fddca193355" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 46851 } } peers { permanent_uuid: "bf9d58c0fe34448c8963c0ca696b3c1b" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44775 } }
I20260808 22:21:53.280373 12325 leader_election.cc:288] T 68719037a0af4cbdbe562b557293b9df P 1fd4acc8159e428bb11f17af176c3542 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 50ce356aabd9458197537fddca193355 (127.9.176.194:46851), bf9d58c0fe34448c8963c0ca696b3c1b (127.9.176.193:44775)
I20260808 22:21:53.283931 12137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:53.280572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.194:46851 (local address 127.9.176.195:38483)
0808 22:21:53.280673 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0808 22:21:53.280689 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 22:21:53.280788 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:21:53.281045 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:21:53.281047 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:21:53.281065 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:21:53.281191 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:21:53.281197 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:21:53.282030 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:21:53.282033 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:21:53.282887 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:21:53.282897 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:21:53.283515 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:21:53.283519 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:21:53.283655 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:53.283674 (+ 19us) client_negotiation.cc:769] Sending connection context
0808 22:21:53.283739 (+ 65us) client_negotiation.cc:240] Negotiation successful
0808 22:21:53.283805 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26,"spinlock_wait_cycles":15232}
I20260808 22:21:53.283929 12314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:53.280658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.195:38483 (local address 127.9.176.194:46851)
0808 22:21:53.280747 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0808 22:21:53.280752 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:21:53.280765 (+ 13us) server_negotiation.cc:407] Connection header received
0808 22:21:53.280883 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:21:53.280886 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:21:53.280953 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:21:53.281037 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:21:53.281318 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:53.281931 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:53.283006 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:53.283411 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:53.283514 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:53.283568 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0808 22:21:53.283762 (+ 194us) server_negotiation.cc:299] Negotiation successful
0808 22:21:53.283827 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260808 22:21:53.284484 12031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68719037a0af4cbdbe562b557293b9df" candidate_uuid: "1fd4acc8159e428bb11f17af176c3542" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50ce356aabd9458197537fddca193355" is_pre_election: true
I20260808 22:21:53.284555 12317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:53.280644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.195:44937 (local address 127.9.176.193:44775)
0808 22:21:53.280747 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0808 22:21:53.280752 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:21:53.281086 (+ 334us) server_negotiation.cc:407] Connection header received
0808 22:21:53.281232 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:21:53.281235 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:21:53.281280 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:21:53.281340 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:21:53.281611 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:53.282131 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:53.283646 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:53.284055 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:53.284166 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:53.284208 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0808 22:21:53.284389 (+ 181us) server_negotiation.cc:299] Negotiation successful
0808 22:21:53.284458 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260808 22:21:53.284555 12333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:53.280570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.193:44775 (local address 127.9.176.195:44937)
0808 22:21:53.280998 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0808 22:21:53.281011 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 22:21:53.281124 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:21:53.281371 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:21:53.281374 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:21:53.281392 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:21:53.281521 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:21:53.281528 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:21:53.282243 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:21:53.282247 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:21:53.283526 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:21:53.283536 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:21:53.284166 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:21:53.284169 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:21:53.284265 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:53.284289 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 22:21:53.284357 (+ 68us) client_negotiation.cc:240] Negotiation successful
0808 22:21:53.284420 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"mutex_wait_us":98,"spinlock_wait_cycles":768,"thread_start_us":109,"threads_started":1}
I20260808 22:21:53.284732 12031 raft_consensus.cc:2454] T 68719037a0af4cbdbe562b557293b9df P 50ce356aabd9458197537fddca193355 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fd4acc8159e428bb11f17af176c3542 in term 0.
I20260808 22:21:53.285141 11811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68719037a0af4cbdbe562b557293b9df" candidate_uuid: "1fd4acc8159e428bb11f17af176c3542" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9d58c0fe34448c8963c0ca696b3c1b" is_pre_election: true
I20260808 22:21:53.285190 12125 leader_election.cc:302] T 68719037a0af4cbdbe562b557293b9df P 1fd4acc8159e428bb11f17af176c3542 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fd4acc8159e428bb11f17af176c3542, 50ce356aabd9458197537fddca193355; no voters:
I20260808 22:21:53.285380 11811 raft_consensus.cc:2454] T 68719037a0af4cbdbe562b557293b9df P bf9d58c0fe34448c8963c0ca696b3c1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fd4acc8159e428bb11f17af176c3542 in term 0.
I20260808 22:21:53.285418 12325 raft_consensus.cc:2781] T 68719037a0af4cbdbe562b557293b9df P 1fd4acc8159e428bb11f17af176c3542 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 22:21:53.285490 12325 raft_consensus.cc:482] T 68719037a0af4cbdbe562b557293b9df P 1fd4acc8159e428bb11f17af176c3542 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 22:21:53.285568 12325 raft_consensus.cc:3037] T 68719037a0af4cbdbe562b557293b9df P 1fd4acc8159e428bb11f17af176c3542 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:21:53.286722 12325 raft_consensus.cc:504] T 68719037a0af4cbdbe562b557293b9df P 1fd4acc8159e428bb11f17af176c3542 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fd4acc8159e428bb11f17af176c3542" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46643 } } peers { permanent_uuid: "50ce356aabd9458197537fddca193355" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 46851 } } peers { permanent_uuid: "bf9d58c0fe34448c8963c0ca696b3c1b" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44775 } }
I20260808 22:21:53.286993 12325 leader_election.cc:288] T 68719037a0af4cbdbe562b557293b9df P 1fd4acc8159e428bb11f17af176c3542 [CANDIDATE]: Term 1 election: Requested vote from peers 50ce356aabd9458197537fddca193355 (127.9.176.194:46851), bf9d58c0fe34448c8963c0ca696b3c1b (127.9.176.193:44775)
I20260808 22:21:53.287307 11811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68719037a0af4cbdbe562b557293b9df" candidate_uuid: "1fd4acc8159e428bb11f17af176c3542" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9d58c0fe34448c8963c0ca696b3c1b"
I20260808 22:21:53.287328 12031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68719037a0af4cbdbe562b557293b9df" candidate_uuid: "1fd4acc8159e428bb11f17af176c3542" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50ce356aabd9458197537fddca193355"
I20260808 22:21:53.287423 11811 raft_consensus.cc:3037] T 68719037a0af4cbdbe562b557293b9df P bf9d58c0fe34448c8963c0ca696b3c1b [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:21:53.287423 12031 raft_consensus.cc:3037] T 68719037a0af4cbdbe562b557293b9df P 50ce356aabd9458197537fddca193355 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:21:53.288559 11811 raft_consensus.cc:2454] T 68719037a0af4cbdbe562b557293b9df P bf9d58c0fe34448c8963c0ca696b3c1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fd4acc8159e428bb11f17af176c3542 in term 1.
I20260808 22:21:53.288859 12128 leader_election.cc:302] T 68719037a0af4cbdbe562b557293b9df P 1fd4acc8159e428bb11f17af176c3542 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fd4acc8159e428bb11f17af176c3542, bf9d58c0fe34448c8963c0ca696b3c1b; no voters:
I20260808 22:21:53.289065 12325 raft_consensus.cc:2781] T 68719037a0af4cbdbe562b557293b9df P 1fd4acc8159e428bb11f17af176c3542 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:21:53.289100 12031 raft_consensus.cc:2454] T 68719037a0af4cbdbe562b557293b9df P 50ce356aabd9458197537fddca193355 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fd4acc8159e428bb11f17af176c3542 in term 1.
I20260808 22:21:53.289270 12325 raft_consensus.cc:686] T 68719037a0af4cbdbe562b557293b9df P 1fd4acc8159e428bb11f17af176c3542 [term 1 LEADER]: Becoming Leader. State: Replica: 1fd4acc8159e428bb11f17af176c3542, State: Running, Role: LEADER
I20260808 22:21:53.289592 12325 consensus_queue.cc:237] T 68719037a0af4cbdbe562b557293b9df P 1fd4acc8159e428bb11f17af176c3542 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fd4acc8159e428bb11f17af176c3542" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46643 } } peers { permanent_uuid: "50ce356aabd9458197537fddca193355" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 46851 } } peers { permanent_uuid: "bf9d58c0fe34448c8963c0ca696b3c1b" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44775 } }
I20260808 22:21:53.292349 11590 catalog_manager.cc:5495] T 68719037a0af4cbdbe562b557293b9df P 1fd4acc8159e428bb11f17af176c3542 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1fd4acc8159e428bb11f17af176c3542 (127.9.176.195). New cstate: current_term: 1 leader_uuid: "1fd4acc8159e428bb11f17af176c3542" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fd4acc8159e428bb11f17af176c3542" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "50ce356aabd9458197537fddca193355" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 46851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf9d58c0fe34448c8963c0ca696b3c1b" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44775 } health_report { overall_health: UNKNOWN } } }
I20260808 22:21:53.313726 12312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:53.310268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58076 (local address 127.9.176.195:46643)
0808 22:21:53.310394 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0808 22:21:53.310398 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:21:53.310411 (+ 13us) server_negotiation.cc:407] Connection header received
0808 22:21:53.310554 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:21:53.310556 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:21:53.310613 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:21:53.310690 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:21:53.310886 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:53.311367 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:53.312217 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:53.312479 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:53.312564 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:53.312820 (+ 256us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:21:53.313272 (+ 452us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:21:53.313355 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0808 22:21:53.313512 (+ 157us) server_negotiation.cc:299] Negotiation successful
0808 22:21:53.313571 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 22:21:53.319522 11811 raft_consensus.cc:1261] T 68719037a0af4cbdbe562b557293b9df P bf9d58c0fe34448c8963c0ca696b3c1b [term 1 FOLLOWER]: Refusing update from remote peer 1fd4acc8159e428bb11f17af176c3542: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:21:53.320165 12325 consensus_queue.cc:1035] T 68719037a0af4cbdbe562b557293b9df P 1fd4acc8159e428bb11f17af176c3542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf9d58c0fe34448c8963c0ca696b3c1b" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:21:53.321818 12031 raft_consensus.cc:1261] T 68719037a0af4cbdbe562b557293b9df P 50ce356aabd9458197537fddca193355 [term 1 FOLLOWER]: Refusing update from remote peer 1fd4acc8159e428bb11f17af176c3542: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:21:53.322397 12336 consensus_queue.cc:1035] T 68719037a0af4cbdbe562b557293b9df P 1fd4acc8159e428bb11f17af176c3542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50ce356aabd9458197537fddca193355" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 46851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:21:53.327418 12341 mvcc.cc:205] Tried to move back new op lower bound from 7316388713749377024 to 7316388713636777984. Current Snapshot: MvccSnapshot[applied={T|T < 7316388713749377024}]
I20260808 22:21:53.328846 12344 mvcc.cc:205] Tried to move back new op lower bound from 7316388713749377024 to 7316388713636777984. Current Snapshot: MvccSnapshot[applied={T|T < 7316388713749377024}]
I20260808 22:21:53.329048 12346 mvcc.cc:205] Tried to move back new op lower bound from 7316388713749377024 to 7316388713636777984. Current Snapshot: MvccSnapshot[applied={T|T < 7316388713749377024}]
I20260808 22:21:53.341363 11588 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60042:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:21:53.341583 11588 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60042:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:21:53.344111 11588 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4533bca6608847efb2cf76f437edff0c: Sending DeleteTablet for 3 replicas of tablet 68719037a0af4cbdbe562b557293b9df
I20260808 22:21:53.344937 12002 tablet_service.cc:1514] Processing DeleteTablet for tablet 68719037a0af4cbdbe562b557293b9df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:21:53 UTC) from {username='kudu', principal='kudu/127.9.176.254@KRBTEST.COM'} at 127.0.0.1:40638
I20260808 22:21:53.344964 12198 tablet_service.cc:1514] Processing DeleteTablet for tablet 68719037a0af4cbdbe562b557293b9df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:21:53 UTC) from {username='kudu', principal='kudu/127.9.176.254@KRBTEST.COM'} at 127.0.0.1:58068
I20260808 22:21:53.345003 11785 tablet_service.cc:1514] Processing DeleteTablet for tablet 68719037a0af4cbdbe562b557293b9df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:21:53 UTC) from {username='kudu', principal='kudu/127.9.176.254@KRBTEST.COM'} at 127.0.0.1:36790
I20260808 22:21:53.345516 12357 tablet_replica.cc:330] T 68719037a0af4cbdbe562b557293b9df P 50ce356aabd9458197537fddca193355: stopping tablet replica
I20260808 22:21:53.345803 12357 raft_consensus.cc:2229] T 68719037a0af4cbdbe562b557293b9df P 50ce356aabd9458197537fddca193355 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:21:53.346091 12357 raft_consensus.cc:2258] T 68719037a0af4cbdbe562b557293b9df P 50ce356aabd9458197537fddca193355 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:21:53.346352 11588 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60042:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 22:21:53.346707 11588 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 22:21:53.347359 12355 tablet_replica.cc:330] T 68719037a0af4cbdbe562b557293b9df P 1fd4acc8159e428bb11f17af176c3542: stopping tablet replica
I20260808 22:21:53.347606 12355 raft_consensus.cc:2229] T 68719037a0af4cbdbe562b557293b9df P 1fd4acc8159e428bb11f17af176c3542 [term 1 LEADER]: Raft consensus shutting down.
I20260808 22:21:53.347972 12355 raft_consensus.cc:2258] T 68719037a0af4cbdbe562b557293b9df P 1fd4acc8159e428bb11f17af176c3542 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:21:53.348493 12356 tablet_replica.cc:330] T 68719037a0af4cbdbe562b557293b9df P bf9d58c0fe34448c8963c0ca696b3c1b: stopping tablet replica
I20260808 22:21:53.348822 12356 raft_consensus.cc:2229] T 68719037a0af4cbdbe562b557293b9df P bf9d58c0fe34448c8963c0ca696b3c1b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:21:53.349189 12356 raft_consensus.cc:2258] T 68719037a0af4cbdbe562b557293b9df P bf9d58c0fe34448c8963c0ca696b3c1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:21:53.349562 12355 ts_tablet_manager.cc:1905] T 68719037a0af4cbdbe562b557293b9df P 1fd4acc8159e428bb11f17af176c3542: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:21:53.350545 12356 ts_tablet_manager.cc:1905] T 68719037a0af4cbdbe562b557293b9df P bf9d58c0fe34448c8963c0ca696b3c1b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:21:53.351240 11785 tablet_service.cc:1467] Processing CreateTablet for tablet c8e16c9346ed4f1b838d65d6b985ea60 (DEFAULT_TABLE table=test-table [id=f97096aa2fe44efebe627eb5acd27401]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:21:53.351420 12198 tablet_service.cc:1467] Processing CreateTablet for tablet c8e16c9346ed4f1b838d65d6b985ea60 (DEFAULT_TABLE table=test-table [id=f97096aa2fe44efebe627eb5acd27401]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:21:53.351665 12198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8e16c9346ed4f1b838d65d6b985ea60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:21:53.351873 11785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8e16c9346ed4f1b838d65d6b985ea60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:21:53.352329 12355 ts_tablet_manager.cc:1918] T 68719037a0af4cbdbe562b557293b9df P 1fd4acc8159e428bb11f17af176c3542: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 22:21:53.352404 12355 log.cc:1196] T 68719037a0af4cbdbe562b557293b9df P 1fd4acc8159e428bb11f17af176c3542: Deleting WAL directory at /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/minicluster-data/ts-2/wal/wals/68719037a0af4cbdbe562b557293b9df
I20260808 22:21:53.352702 12355 ts_tablet_manager.cc:1939] T 68719037a0af4cbdbe562b557293b9df P 1fd4acc8159e428bb11f17af176c3542: Deleting consensus metadata
I20260808 22:21:53.353984 12319 tablet_bootstrap.cc:492] T c8e16c9346ed4f1b838d65d6b985ea60 P 1fd4acc8159e428bb11f17af176c3542: Bootstrap starting.
I20260808 22:21:53.354501 12322 tablet_bootstrap.cc:492] T c8e16c9346ed4f1b838d65d6b985ea60 P bf9d58c0fe34448c8963c0ca696b3c1b: Bootstrap starting.
I20260808 22:21:53.355049 12319 tablet_bootstrap.cc:654] T c8e16c9346ed4f1b838d65d6b985ea60 P 1fd4acc8159e428bb11f17af176c3542: Neither blocks nor log segments found. Creating new log.
I20260808 22:21:53.355644 12322 tablet_bootstrap.cc:654] T c8e16c9346ed4f1b838d65d6b985ea60 P bf9d58c0fe34448c8963c0ca696b3c1b: Neither blocks nor log segments found. Creating new log.
I20260808 22:21:53.355947 11568 catalog_manager.cc:4841] TS 1fd4acc8159e428bb11f17af176c3542 (127.9.176.195:46643): tablet 68719037a0af4cbdbe562b557293b9df (table test-table [id=0bf2c5b3e4bf4ec9ad583f6fb2181932]) successfully deleted
I20260808 22:21:53.356784 12002 tablet_service.cc:1467] Processing CreateTablet for tablet c8e16c9346ed4f1b838d65d6b985ea60 (DEFAULT_TABLE table=test-table [id=f97096aa2fe44efebe627eb5acd27401]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:21:53.357126 12002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8e16c9346ed4f1b838d65d6b985ea60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:21:53.357234 12322 tablet_bootstrap.cc:492] T c8e16c9346ed4f1b838d65d6b985ea60 P bf9d58c0fe34448c8963c0ca696b3c1b: No bootstrap required, opened a new log
I20260808 22:21:53.357321 12322 ts_tablet_manager.cc:1397] T c8e16c9346ed4f1b838d65d6b985ea60 P bf9d58c0fe34448c8963c0ca696b3c1b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 22:21:53.357873 12356 ts_tablet_manager.cc:1918] T 68719037a0af4cbdbe562b557293b9df P bf9d58c0fe34448c8963c0ca696b3c1b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 22:21:53.357969 12356 log.cc:1196] T 68719037a0af4cbdbe562b557293b9df P bf9d58c0fe34448c8963c0ca696b3c1b: Deleting WAL directory at /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/minicluster-data/ts-0/wal/wals/68719037a0af4cbdbe562b557293b9df
I20260808 22:21:53.358068 12322 raft_consensus.cc:348] T c8e16c9346ed4f1b838d65d6b985ea60 P bf9d58c0fe34448c8963c0ca696b3c1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9d58c0fe34448c8963c0ca696b3c1b" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44775 } } peers { permanent_uuid: "50ce356aabd9458197537fddca193355" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 46851 } } peers { permanent_uuid: "1fd4acc8159e428bb11f17af176c3542" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46643 } }
I20260808 22:21:53.358207 12322 raft_consensus.cc:374] T c8e16c9346ed4f1b838d65d6b985ea60 P bf9d58c0fe34448c8963c0ca696b3c1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:21:53.358251 12322 raft_consensus.cc:729] T c8e16c9346ed4f1b838d65d6b985ea60 P bf9d58c0fe34448c8963c0ca696b3c1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9d58c0fe34448c8963c0ca696b3c1b, State: Initialized, Role: FOLLOWER
I20260808 22:21:53.358317 12356 ts_tablet_manager.cc:1939] T 68719037a0af4cbdbe562b557293b9df P bf9d58c0fe34448c8963c0ca696b3c1b: Deleting consensus metadata
I20260808 22:21:53.358393 12322 consensus_queue.cc:260] T c8e16c9346ed4f1b838d65d6b985ea60 P bf9d58c0fe34448c8963c0ca696b3c1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9d58c0fe34448c8963c0ca696b3c1b" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44775 } } peers { permanent_uuid: "50ce356aabd9458197537fddca193355" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 46851 } } peers { permanent_uuid: "1fd4acc8159e428bb11f17af176c3542" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46643 } }
I20260808 22:21:53.358687 12322 ts_tablet_manager.cc:1428] T c8e16c9346ed4f1b838d65d6b985ea60 P bf9d58c0fe34448c8963c0ca696b3c1b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260808 22:21:53.360664 11572 catalog_manager.cc:4841] TS bf9d58c0fe34448c8963c0ca696b3c1b (127.9.176.193:44775): tablet 68719037a0af4cbdbe562b557293b9df (table test-table [id=0bf2c5b3e4bf4ec9ad583f6fb2181932]) successfully deleted
I20260808 22:21:53.361181 12320 tablet_bootstrap.cc:492] T c8e16c9346ed4f1b838d65d6b985ea60 P 50ce356aabd9458197537fddca193355: Bootstrap starting.
I20260808 22:21:53.362197 12320 tablet_bootstrap.cc:654] T c8e16c9346ed4f1b838d65d6b985ea60 P 50ce356aabd9458197537fddca193355: Neither blocks nor log segments found. Creating new log.
I20260808 22:21:53.363473 12320 tablet_bootstrap.cc:492] T c8e16c9346ed4f1b838d65d6b985ea60 P 50ce356aabd9458197537fddca193355: No bootstrap required, opened a new log
I20260808 22:21:53.363562 12320 ts_tablet_manager.cc:1397] T c8e16c9346ed4f1b838d65d6b985ea60 P 50ce356aabd9458197537fddca193355: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260808 22:21:53.363996 12320 raft_consensus.cc:348] T c8e16c9346ed4f1b838d65d6b985ea60 P 50ce356aabd9458197537fddca193355 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9d58c0fe34448c8963c0ca696b3c1b" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44775 } } peers { permanent_uuid: "50ce356aabd9458197537fddca193355" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 46851 } } peers { permanent_uuid: "1fd4acc8159e428bb11f17af176c3542" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46643 } }
I20260808 22:21:53.364094 12320 raft_consensus.cc:374] T c8e16c9346ed4f1b838d65d6b985ea60 P 50ce356aabd9458197537fddca193355 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:21:53.364125 12320 raft_consensus.cc:729] T c8e16c9346ed4f1b838d65d6b985ea60 P 50ce356aabd9458197537fddca193355 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50ce356aabd9458197537fddca193355, State: Initialized, Role: FOLLOWER
I20260808 22:21:53.364239 12320 consensus_queue.cc:260] T c8e16c9346ed4f1b838d65d6b985ea60 P 50ce356aabd9458197537fddca193355 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9d58c0fe34448c8963c0ca696b3c1b" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44775 } } peers { permanent_uuid: "50ce356aabd9458197537fddca193355" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 46851 } } peers { permanent_uuid: "1fd4acc8159e428bb11f17af176c3542" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46643 } }
I20260808 22:21:53.364467 12320 ts_tablet_manager.cc:1428] T c8e16c9346ed4f1b838d65d6b985ea60 P 50ce356aabd9458197537fddca193355: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260808 22:21:53.365043 12357 ts_tablet_manager.cc:1905] T 68719037a0af4cbdbe562b557293b9df P 50ce356aabd9458197537fddca193355: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:21:53.367767 12357 ts_tablet_manager.cc:1918] T 68719037a0af4cbdbe562b557293b9df P 50ce356aabd9458197537fddca193355: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 22:21:53.367873 12357 log.cc:1196] T 68719037a0af4cbdbe562b557293b9df P 50ce356aabd9458197537fddca193355: Deleting WAL directory at /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/minicluster-data/ts-1/wal/wals/68719037a0af4cbdbe562b557293b9df
I20260808 22:21:53.368202 12357 ts_tablet_manager.cc:1939] T 68719037a0af4cbdbe562b557293b9df P 50ce356aabd9458197537fddca193355: Deleting consensus metadata
I20260808 22:21:53.369871 11567 catalog_manager.cc:4841] TS 50ce356aabd9458197537fddca193355 (127.9.176.194:46851): tablet 68719037a0af4cbdbe562b557293b9df (table test-table [id=0bf2c5b3e4bf4ec9ad583f6fb2181932]) successfully deleted
I20260808 22:21:53.372496 12319 tablet_bootstrap.cc:492] T c8e16c9346ed4f1b838d65d6b985ea60 P 1fd4acc8159e428bb11f17af176c3542: No bootstrap required, opened a new log
I20260808 22:21:53.372588 12319 ts_tablet_manager.cc:1397] T c8e16c9346ed4f1b838d65d6b985ea60 P 1fd4acc8159e428bb11f17af176c3542: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.000s
I20260808 22:21:53.373031 12319 raft_consensus.cc:348] T c8e16c9346ed4f1b838d65d6b985ea60 P 1fd4acc8159e428bb11f17af176c3542 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9d58c0fe34448c8963c0ca696b3c1b" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44775 } } peers { permanent_uuid: "50ce356aabd9458197537fddca193355" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 46851 } } peers { permanent_uuid: "1fd4acc8159e428bb11f17af176c3542" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46643 } }
I20260808 22:21:53.373140 12319 raft_consensus.cc:374] T c8e16c9346ed4f1b838d65d6b985ea60 P 1fd4acc8159e428bb11f17af176c3542 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:21:53.373172 12319 raft_consensus.cc:729] T c8e16c9346ed4f1b838d65d6b985ea60 P 1fd4acc8159e428bb11f17af176c3542 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fd4acc8159e428bb11f17af176c3542, State: Initialized, Role: FOLLOWER
I20260808 22:21:53.373291 12319 consensus_queue.cc:260] T c8e16c9346ed4f1b838d65d6b985ea60 P 1fd4acc8159e428bb11f17af176c3542 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9d58c0fe34448c8963c0ca696b3c1b" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44775 } } peers { permanent_uuid: "50ce356aabd9458197537fddca193355" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 46851 } } peers { permanent_uuid: "1fd4acc8159e428bb11f17af176c3542" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46643 } }
I20260808 22:21:53.373509 12319 ts_tablet_manager.cc:1428] T c8e16c9346ed4f1b838d65d6b985ea60 P 1fd4acc8159e428bb11f17af176c3542: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 22:21:53.388751 12329 raft_consensus.cc:482] T c8e16c9346ed4f1b838d65d6b985ea60 P bf9d58c0fe34448c8963c0ca696b3c1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 22:21:53.388871 12329 raft_consensus.cc:504] T c8e16c9346ed4f1b838d65d6b985ea60 P bf9d58c0fe34448c8963c0ca696b3c1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9d58c0fe34448c8963c0ca696b3c1b" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44775 } } peers { permanent_uuid: "50ce356aabd9458197537fddca193355" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 46851 } } peers { permanent_uuid: "1fd4acc8159e428bb11f17af176c3542" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46643 } }
I20260808 22:21:53.390157 12329 leader_election.cc:288] T c8e16c9346ed4f1b838d65d6b985ea60 P bf9d58c0fe34448c8963c0ca696b3c1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 50ce356aabd9458197537fddca193355 (127.9.176.194:46851), 1fd4acc8159e428bb11f17af176c3542 (127.9.176.195:46643)
I20260808 22:21:53.395198 12312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:53.390850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.193:47667 (local address 127.9.176.195:46643)
0808 22:21:53.390975 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0808 22:21:53.390979 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:21:53.390990 (+ 11us) server_negotiation.cc:407] Connection header received
0808 22:21:53.391031 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:21:53.391034 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:21:53.391079 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:21:53.391157 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:21:53.391576 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:53.392309 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:53.393897 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:53.394498 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:53.394581 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:53.394644 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0808 22:21:53.394994 (+ 350us) server_negotiation.cc:299] Negotiation successful
0808 22:21:53.395072 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 22:21:53.395479 12314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:53.390991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.193:56325 (local address 127.9.176.194:46851)
0808 22:21:53.391136 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0808 22:21:53.391140 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:21:53.391153 (+ 13us) server_negotiation.cc:407] Connection header received
0808 22:21:53.391200 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:21:53.391203 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:21:53.391243 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:21:53.391304 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:21:53.392051 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:53.392745 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:53.394291 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:53.394838 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:53.394920 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:53.394967 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0808 22:21:53.395291 (+ 324us) server_negotiation.cc:299] Negotiation successful
0808 22:21:53.395373 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 22:21:53.395849 12363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:53.390311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.194:46851 (local address 127.9.176.193:56325)
0808 22:21:53.390550 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0808 22:21:53.390564 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 22:21:53.390668 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:21:53.391593 (+ 925us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:21:53.391595 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:21:53.391615 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:21:53.391914 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:21:53.391923 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:21:53.392888 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:21:53.392917 (+ 29us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:21:53.394161 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:21:53.394170 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:21:53.395048 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:21:53.395051 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:21:53.395151 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:53.395172 (+ 21us) client_negotiation.cc:769] Sending connection context
0808 22:21:53.395612 (+ 440us) client_negotiation.cc:240] Negotiation successful
0808 22:21:53.395679 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":60,"threads_started":1}
I20260808 22:21:53.395856 12364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:53.390421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.195:46643 (local address 127.9.176.193:47667)
0808 22:21:53.390674 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0808 22:21:53.390685 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 22:21:53.390752 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:21:53.391274 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:21:53.391277 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:21:53.391297 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:21:53.391472 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:21:53.391483 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:21:53.392430 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:21:53.392434 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:21:53.393757 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:21:53.393768 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:21:53.394753 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:21:53.394757 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:21:53.394855 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:53.394881 (+ 26us) client_negotiation.cc:769] Sending connection context
0808 22:21:53.395488 (+ 607us) client_negotiation.cc:240] Negotiation successful
0808 22:21:53.395683 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"mutex_wait_us":1,"thread_start_us":107,"threads_started":1}
I20260808 22:21:53.396281 12031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8e16c9346ed4f1b838d65d6b985ea60" candidate_uuid: "bf9d58c0fe34448c8963c0ca696b3c1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50ce356aabd9458197537fddca193355" is_pre_election: true
I20260808 22:21:53.396436 12031 raft_consensus.cc:2454] T c8e16c9346ed4f1b838d65d6b985ea60 P 50ce356aabd9458197537fddca193355 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf9d58c0fe34448c8963c0ca696b3c1b in term 0.
I20260808 22:21:53.396571 12220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8e16c9346ed4f1b838d65d6b985ea60" candidate_uuid: "bf9d58c0fe34448c8963c0ca696b3c1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fd4acc8159e428bb11f17af176c3542" is_pre_election: true
I20260808 22:21:53.396811 12220 raft_consensus.cc:2454] T c8e16c9346ed4f1b838d65d6b985ea60 P 1fd4acc8159e428bb11f17af176c3542 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf9d58c0fe34448c8963c0ca696b3c1b in term 0.
I20260808 22:21:53.396847 11710 leader_election.cc:302] T c8e16c9346ed4f1b838d65d6b985ea60 P bf9d58c0fe34448c8963c0ca696b3c1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50ce356aabd9458197537fddca193355, bf9d58c0fe34448c8963c0ca696b3c1b; no voters:
I20260808 22:21:53.397140 12329 raft_consensus.cc:2781] T c8e16c9346ed4f1b838d65d6b985ea60 P bf9d58c0fe34448c8963c0ca696b3c1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 22:21:53.397198 12329 raft_consensus.cc:482] T c8e16c9346ed4f1b838d65d6b985ea60 P bf9d58c0fe34448c8963c0ca696b3c1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 22:21:53.397231 12329 raft_consensus.cc:3037] T c8e16c9346ed4f1b838d65d6b985ea60 P bf9d58c0fe34448c8963c0ca696b3c1b [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:21:53.398278 12329 raft_consensus.cc:504] T c8e16c9346ed4f1b838d65d6b985ea60 P bf9d58c0fe34448c8963c0ca696b3c1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9d58c0fe34448c8963c0ca696b3c1b" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44775 } } peers { permanent_uuid: "50ce356aabd9458197537fddca193355" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 46851 } } peers { permanent_uuid: "1fd4acc8159e428bb11f17af176c3542" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46643 } }
I20260808 22:21:53.398603 12329 leader_election.cc:288] T c8e16c9346ed4f1b838d65d6b985ea60 P bf9d58c0fe34448c8963c0ca696b3c1b [CANDIDATE]: Term 1 election: Requested vote from peers 50ce356aabd9458197537fddca193355 (127.9.176.194:46851), 1fd4acc8159e428bb11f17af176c3542 (127.9.176.195:46643)
I20260808 22:21:53.399034 12031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8e16c9346ed4f1b838d65d6b985ea60" candidate_uuid: "bf9d58c0fe34448c8963c0ca696b3c1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50ce356aabd9458197537fddca193355"
I20260808 22:21:53.399101 12220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8e16c9346ed4f1b838d65d6b985ea60" candidate_uuid: "bf9d58c0fe34448c8963c0ca696b3c1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fd4acc8159e428bb11f17af176c3542"
I20260808 22:21:53.399163 12031 raft_consensus.cc:3037] T c8e16c9346ed4f1b838d65d6b985ea60 P 50ce356aabd9458197537fddca193355 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:21:53.399206 12220 raft_consensus.cc:3037] T c8e16c9346ed4f1b838d65d6b985ea60 P 1fd4acc8159e428bb11f17af176c3542 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:21:53.400112 12220 raft_consensus.cc:2454] T c8e16c9346ed4f1b838d65d6b985ea60 P 1fd4acc8159e428bb11f17af176c3542 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf9d58c0fe34448c8963c0ca696b3c1b in term 1.
I20260808 22:21:53.400282 12031 raft_consensus.cc:2454] T c8e16c9346ed4f1b838d65d6b985ea60 P 50ce356aabd9458197537fddca193355 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf9d58c0fe34448c8963c0ca696b3c1b in term 1.
I20260808 22:21:53.400374 11711 leader_election.cc:302] T c8e16c9346ed4f1b838d65d6b985ea60 P bf9d58c0fe34448c8963c0ca696b3c1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fd4acc8159e428bb11f17af176c3542, bf9d58c0fe34448c8963c0ca696b3c1b; no voters:
I20260808 22:21:53.400542 12329 raft_consensus.cc:2781] T c8e16c9346ed4f1b838d65d6b985ea60 P bf9d58c0fe34448c8963c0ca696b3c1b [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:21:53.400764 12329 raft_consensus.cc:686] T c8e16c9346ed4f1b838d65d6b985ea60 P bf9d58c0fe34448c8963c0ca696b3c1b [term 1 LEADER]: Becoming Leader. State: Replica: bf9d58c0fe34448c8963c0ca696b3c1b, State: Running, Role: LEADER
I20260808 22:21:53.401047 12329 consensus_queue.cc:237] T c8e16c9346ed4f1b838d65d6b985ea60 P bf9d58c0fe34448c8963c0ca696b3c1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9d58c0fe34448c8963c0ca696b3c1b" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44775 } } peers { permanent_uuid: "50ce356aabd9458197537fddca193355" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 46851 } } peers { permanent_uuid: "1fd4acc8159e428bb11f17af176c3542" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46643 } }
I20260808 22:21:53.403609 11590 catalog_manager.cc:5495] T c8e16c9346ed4f1b838d65d6b985ea60 P bf9d58c0fe34448c8963c0ca696b3c1b reported cstate change: term changed from 0 to 1, leader changed from <none> to bf9d58c0fe34448c8963c0ca696b3c1b (127.9.176.193). New cstate: current_term: 1 leader_uuid: "bf9d58c0fe34448c8963c0ca696b3c1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9d58c0fe34448c8963c0ca696b3c1b" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "50ce356aabd9458197537fddca193355" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 46851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fd4acc8159e428bb11f17af176c3542" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46643 } health_report { overall_health: UNKNOWN } } }
W20260808 22:21:53.436228 12288 tablet.cc:2369] T c8e16c9346ed4f1b838d65d6b985ea60 P 1fd4acc8159e428bb11f17af176c3542: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 08 22:21:53 dist-test-slave-ncp9 krb5kdc[11523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227712, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.254@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
I20260808 22:21:53.447755 11657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:53.438739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60046 (local address 127.9.176.254:33915)
0808 22:21:53.438855 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0808 22:21:53.438860 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:21:53.439057 (+ 197us) server_negotiation.cc:407] Connection header received
0808 22:21:53.439166 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:21:53.439169 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:21:53.439206 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:21:53.439269 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:21:53.439925 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:53.440441 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:53.441353 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:53.441562 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:53.441645 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:53.444212 (+ 2567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:21:53.444217 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:21:53.444220 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:21:53.444250 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:21:53.445917 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:21:53.446437 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:21:53.446440 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:21:53.446442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:21:53.446529 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:21:53.446816 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:21:53.446819 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:21:53.446821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:21:53.447036 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:21:53.447128 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0808 22:21:53.447466 (+ 338us) server_negotiation.cc:299] Negotiation successful
0808 22:21:53.447594 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260808 22:21:53.448450 12373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:53.438697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:33915 (local address 127.0.0.1:60046)
0808 22:21:53.438968 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0808 22:21:53.438979 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 22:21:53.439090 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:21:53.439300 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:21:53.439303 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:21:53.439717 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:21:53.439814 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:21:53.439821 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:21:53.440547 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:21:53.440550 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:21:53.441250 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:21:53.441264 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:21:53.441654 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:21:53.441657 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:21:53.441730 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:53.442266 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:21:53.442278 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:21:53.444071 (+ 1793us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:21:53.446029 (+ 1958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:21:53.446039 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:21:53.446046 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:21:53.446332 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:21:53.446622 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:21:53.446625 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:21:53.446627 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:21:53.446719 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:21:53.447173 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:21:53.447179 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:21:53.447296 (+ 117us) client_negotiation.cc:769] Sending connection context
0808 22:21:53.448176 (+ 880us) client_negotiation.cc:240] Negotiation successful
0808 22:21:53.448301 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":106,"threads_started":1}
I20260808 22:21:53.453346 11590 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.9.176.254@KRBTEST.COM'} at 127.0.0.1:60046:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260808 22:21:53.454152 11590 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 22:21:53.459106 11785 tablet_service.cc:1467] Processing CreateTablet for tablet 389759e9386e478a8bda705033c55cf9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cfabc0c293fb418da46736175f806bcf]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 22:21:53.459548 11785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 389759e9386e478a8bda705033c55cf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:21:53.459421 12002 tablet_service.cc:1467] Processing CreateTablet for tablet 389759e9386e478a8bda705033c55cf9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cfabc0c293fb418da46736175f806bcf]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 22:21:53.459774 12002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 389759e9386e478a8bda705033c55cf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:21:53.461290 12198 tablet_service.cc:1467] Processing CreateTablet for tablet 389759e9386e478a8bda705033c55cf9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cfabc0c293fb418da46736175f806bcf]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 22:21:53.461650 12198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 389759e9386e478a8bda705033c55cf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:21:53.462504 12322 tablet_bootstrap.cc:492] T 389759e9386e478a8bda705033c55cf9 P bf9d58c0fe34448c8963c0ca696b3c1b: Bootstrap starting.
I20260808 22:21:53.462787 12320 tablet_bootstrap.cc:492] T 389759e9386e478a8bda705033c55cf9 P 50ce356aabd9458197537fddca193355: Bootstrap starting.
I20260808 22:21:53.463451 12322 tablet_bootstrap.cc:654] T 389759e9386e478a8bda705033c55cf9 P bf9d58c0fe34448c8963c0ca696b3c1b: Neither blocks nor log segments found. Creating new log.
I20260808 22:21:53.463716 12320 tablet_bootstrap.cc:654] T 389759e9386e478a8bda705033c55cf9 P 50ce356aabd9458197537fddca193355: Neither blocks nor log segments found. Creating new log.
I20260808 22:21:53.464358 12319 tablet_bootstrap.cc:492] T 389759e9386e478a8bda705033c55cf9 P 1fd4acc8159e428bb11f17af176c3542: Bootstrap starting.
I20260808 22:21:53.464912 12322 tablet_bootstrap.cc:492] T 389759e9386e478a8bda705033c55cf9 P bf9d58c0fe34448c8963c0ca696b3c1b: No bootstrap required, opened a new log
I20260808 22:21:53.464998 12322 ts_tablet_manager.cc:1397] T 389759e9386e478a8bda705033c55cf9 P bf9d58c0fe34448c8963c0ca696b3c1b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 22:21:53.465453 12319 tablet_bootstrap.cc:654] T 389759e9386e478a8bda705033c55cf9 P 1fd4acc8159e428bb11f17af176c3542: Neither blocks nor log segments found. Creating new log.
I20260808 22:21:53.465430 12322 raft_consensus.cc:348] T 389759e9386e478a8bda705033c55cf9 P bf9d58c0fe34448c8963c0ca696b3c1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9d58c0fe34448c8963c0ca696b3c1b" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44775 } } peers { permanent_uuid: "50ce356aabd9458197537fddca193355" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 46851 } } peers { permanent_uuid: "1fd4acc8159e428bb11f17af176c3542" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46643 } }
I20260808 22:21:53.465538 12322 raft_consensus.cc:374] T 389759e9386e478a8bda705033c55cf9 P bf9d58c0fe34448c8963c0ca696b3c1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:21:53.465569 12322 raft_consensus.cc:729] T 389759e9386e478a8bda705033c55cf9 P bf9d58c0fe34448c8963c0ca696b3c1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9d58c0fe34448c8963c0ca696b3c1b, State: Initialized, Role: FOLLOWER
I20260808 22:21:53.465705 12322 consensus_queue.cc:260] T 389759e9386e478a8bda705033c55cf9 P bf9d58c0fe34448c8963c0ca696b3c1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9d58c0fe34448c8963c0ca696b3c1b" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44775 } } peers { permanent_uuid: "50ce356aabd9458197537fddca193355" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 46851 } } peers { permanent_uuid: "1fd4acc8159e428bb11f17af176c3542" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46643 } }
I20260808 22:21:53.465991 12322 ts_tablet_manager.cc:1428] T 389759e9386e478a8bda705033c55cf9 P bf9d58c0fe34448c8963c0ca696b3c1b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260808 22:21:53.466014 12329 tablet_replica.cc:439] T 389759e9386e478a8bda705033c55cf9 P bf9d58c0fe34448c8963c0ca696b3c1b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9d58c0fe34448c8963c0ca696b3c1b" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44775 } } peers { permanent_uuid: "50ce356aabd9458197537fddca193355" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 46851 } } peers { permanent_uuid: "1fd4acc8159e428bb11f17af176c3542" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46643 } } }
I20260808 22:21:53.466149 12329 tablet_replica.cc:442] T 389759e9386e478a8bda705033c55cf9 P bf9d58c0fe34448c8963c0ca696b3c1b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 22:21:53.467195 12319 tablet_bootstrap.cc:492] T 389759e9386e478a8bda705033c55cf9 P 1fd4acc8159e428bb11f17af176c3542: No bootstrap required, opened a new log
I20260808 22:21:53.467271 12319 ts_tablet_manager.cc:1397] T 389759e9386e478a8bda705033c55cf9 P 1fd4acc8159e428bb11f17af176c3542: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.003s
I20260808 22:21:53.467506 12320 tablet_bootstrap.cc:492] T 389759e9386e478a8bda705033c55cf9 P 50ce356aabd9458197537fddca193355: No bootstrap required, opened a new log
I20260808 22:21:53.467602 12320 ts_tablet_manager.cc:1397] T 389759e9386e478a8bda705033c55cf9 P 50ce356aabd9458197537fddca193355: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20260808 22:21:53.467648 12319 raft_consensus.cc:348] T 389759e9386e478a8bda705033c55cf9 P 1fd4acc8159e428bb11f17af176c3542 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9d58c0fe34448c8963c0ca696b3c1b" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44775 } } peers { permanent_uuid: "50ce356aabd9458197537fddca193355" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 46851 } } peers { permanent_uuid: "1fd4acc8159e428bb11f17af176c3542" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46643 } }
I20260808 22:21:53.467756 12319 raft_consensus.cc:374] T 389759e9386e478a8bda705033c55cf9 P 1fd4acc8159e428bb11f17af176c3542 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:21:53.467789 12319 raft_consensus.cc:729] T 389759e9386e478a8bda705033c55cf9 P 1fd4acc8159e428bb11f17af176c3542 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fd4acc8159e428bb11f17af176c3542, State: Initialized, Role: FOLLOWER
I20260808 22:21:53.467912 12319 consensus_queue.cc:260] T 389759e9386e478a8bda705033c55cf9 P 1fd4acc8159e428bb11f17af176c3542 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9d58c0fe34448c8963c0ca696b3c1b" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44775 } } peers { permanent_uuid: "50ce356aabd9458197537fddca193355" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 46851 } } peers { permanent_uuid: "1fd4acc8159e428bb11f17af176c3542" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46643 } }
I20260808 22:21:53.468036 12320 raft_consensus.cc:348] T 389759e9386e478a8bda705033c55cf9 P 50ce356aabd9458197537fddca193355 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9d58c0fe34448c8963c0ca696b3c1b" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44775 } } peers { permanent_uuid: "50ce356aabd9458197537fddca193355" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 46851 } } peers { permanent_uuid: "1fd4acc8159e428bb11f17af176c3542" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46643 } }
I20260808 22:21:53.468180 12320 raft_consensus.cc:374] T 389759e9386e478a8bda705033c55cf9 P 50ce356aabd9458197537fddca193355 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:21:53.468214 12320 raft_consensus.cc:729] T 389759e9386e478a8bda705033c55cf9 P 50ce356aabd9458197537fddca193355 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50ce356aabd9458197537fddca193355, State: Initialized, Role: FOLLOWER
I20260808 22:21:53.468182 12349 tablet_replica.cc:439] T 389759e9386e478a8bda705033c55cf9 P 1fd4acc8159e428bb11f17af176c3542: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9d58c0fe34448c8963c0ca696b3c1b" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44775 } } peers { permanent_uuid: "50ce356aabd9458197537fddca193355" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 46851 } } peers { permanent_uuid: "1fd4acc8159e428bb11f17af176c3542" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46643 } } }
I20260808 22:21:53.468344 12349 tablet_replica.cc:442] T 389759e9386e478a8bda705033c55cf9 P 1fd4acc8159e428bb11f17af176c3542: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 22:21:53.468324 12320 consensus_queue.cc:260] T 389759e9386e478a8bda705033c55cf9 P 50ce356aabd9458197537fddca193355 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9d58c0fe34448c8963c0ca696b3c1b" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44775 } } peers { permanent_uuid: "50ce356aabd9458197537fddca193355" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 46851 } } peers { permanent_uuid: "1fd4acc8159e428bb11f17af176c3542" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46643 } }
I20260808 22:21:53.468568 12320 ts_tablet_manager.cc:1428] T 389759e9386e478a8bda705033c55cf9 P 50ce356aabd9458197537fddca193355: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260808 22:21:53.468169 12319 ts_tablet_manager.cc:1428] T 389759e9386e478a8bda705033c55cf9 P 1fd4acc8159e428bb11f17af176c3542: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260808 22:21:53.469370 12330 tablet_replica.cc:439] T 389759e9386e478a8bda705033c55cf9 P 50ce356aabd9458197537fddca193355: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9d58c0fe34448c8963c0ca696b3c1b" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44775 } } peers { permanent_uuid: "50ce356aabd9458197537fddca193355" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 46851 } } peers { permanent_uuid: "1fd4acc8159e428bb11f17af176c3542" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46643 } } }
I20260808 22:21:53.469543 12330 tablet_replica.cc:442] T 389759e9386e478a8bda705033c55cf9 P 50ce356aabd9458197537fddca193355: This TxnStatusTablet replica's current role is: FOLLOWER
W20260808 22:21:53.476974 12096 tablet.cc:2369] T 389759e9386e478a8bda705033c55cf9 P 50ce356aabd9458197537fddca193355: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 22:21:53.516201 11889 tablet.cc:2369] T 389759e9386e478a8bda705033c55cf9 P bf9d58c0fe34448c8963c0ca696b3c1b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 22:21:53.557364 12349 raft_consensus.cc:482] T 389759e9386e478a8bda705033c55cf9 P 1fd4acc8159e428bb11f17af176c3542 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 22:21:53.557493 12349 raft_consensus.cc:504] T 389759e9386e478a8bda705033c55cf9 P 1fd4acc8159e428bb11f17af176c3542 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9d58c0fe34448c8963c0ca696b3c1b" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44775 } } peers { permanent_uuid: "50ce356aabd9458197537fddca193355" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 46851 } } peers { permanent_uuid: "1fd4acc8159e428bb11f17af176c3542" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46643 } }
I20260808 22:21:53.557869 12349 leader_election.cc:288] T 389759e9386e478a8bda705033c55cf9 P 1fd4acc8159e428bb11f17af176c3542 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf9d58c0fe34448c8963c0ca696b3c1b (127.9.176.193:44775), 50ce356aabd9458197537fddca193355 (127.9.176.194:46851)
I20260808 22:21:53.558306 11811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "389759e9386e478a8bda705033c55cf9" candidate_uuid: "1fd4acc8159e428bb11f17af176c3542" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9d58c0fe34448c8963c0ca696b3c1b" is_pre_election: true
I20260808 22:21:53.558305 12031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "389759e9386e478a8bda705033c55cf9" candidate_uuid: "1fd4acc8159e428bb11f17af176c3542" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50ce356aabd9458197537fddca193355" is_pre_election: true
I20260808 22:21:53.558465 12031 raft_consensus.cc:2454] T 389759e9386e478a8bda705033c55cf9 P 50ce356aabd9458197537fddca193355 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fd4acc8159e428bb11f17af176c3542 in term 0.
I20260808 22:21:53.558465 11811 raft_consensus.cc:2454] T 389759e9386e478a8bda705033c55cf9 P bf9d58c0fe34448c8963c0ca696b3c1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fd4acc8159e428bb11f17af176c3542 in term 0.
I20260808 22:21:53.558769 12125 leader_election.cc:302] T 389759e9386e478a8bda705033c55cf9 P 1fd4acc8159e428bb11f17af176c3542 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fd4acc8159e428bb11f17af176c3542, 50ce356aabd9458197537fddca193355; no voters:
I20260808 22:21:53.558955 12349 raft_consensus.cc:2781] T 389759e9386e478a8bda705033c55cf9 P 1fd4acc8159e428bb11f17af176c3542 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 22:21:53.559024 12349 raft_consensus.cc:482] T 389759e9386e478a8bda705033c55cf9 P 1fd4acc8159e428bb11f17af176c3542 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 22:21:53.559051 12349 raft_consensus.cc:3037] T 389759e9386e478a8bda705033c55cf9 P 1fd4acc8159e428bb11f17af176c3542 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:21:53.560063 12349 raft_consensus.cc:504] T 389759e9386e478a8bda705033c55cf9 P 1fd4acc8159e428bb11f17af176c3542 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9d58c0fe34448c8963c0ca696b3c1b" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44775 } } peers { permanent_uuid: "50ce356aabd9458197537fddca193355" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 46851 } } peers { permanent_uuid: "1fd4acc8159e428bb11f17af176c3542" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46643 } }
I20260808 22:21:53.560379 12349 leader_election.cc:288] T 389759e9386e478a8bda705033c55cf9 P 1fd4acc8159e428bb11f17af176c3542 [CANDIDATE]: Term 1 election: Requested vote from peers bf9d58c0fe34448c8963c0ca696b3c1b (127.9.176.193:44775), 50ce356aabd9458197537fddca193355 (127.9.176.194:46851)
I20260808 22:21:53.560706 12031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "389759e9386e478a8bda705033c55cf9" candidate_uuid: "1fd4acc8159e428bb11f17af176c3542" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50ce356aabd9458197537fddca193355"
I20260808 22:21:53.560715 11811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "389759e9386e478a8bda705033c55cf9" candidate_uuid: "1fd4acc8159e428bb11f17af176c3542" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9d58c0fe34448c8963c0ca696b3c1b"
I20260808 22:21:53.560812 12031 raft_consensus.cc:3037] T 389759e9386e478a8bda705033c55cf9 P 50ce356aabd9458197537fddca193355 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:21:53.560824 11811 raft_consensus.cc:3037] T 389759e9386e478a8bda705033c55cf9 P bf9d58c0fe34448c8963c0ca696b3c1b [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:21:53.561707 12031 raft_consensus.cc:2454] T 389759e9386e478a8bda705033c55cf9 P 50ce356aabd9458197537fddca193355 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fd4acc8159e428bb11f17af176c3542 in term 1.
I20260808 22:21:53.561939 11811 raft_consensus.cc:2454] T 389759e9386e478a8bda705033c55cf9 P bf9d58c0fe34448c8963c0ca696b3c1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fd4acc8159e428bb11f17af176c3542 in term 1.
I20260808 22:21:53.562003 12125 leader_election.cc:302] T 389759e9386e478a8bda705033c55cf9 P 1fd4acc8159e428bb11f17af176c3542 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fd4acc8159e428bb11f17af176c3542, 50ce356aabd9458197537fddca193355; no voters:
I20260808 22:21:53.562215 12349 raft_consensus.cc:2781] T 389759e9386e478a8bda705033c55cf9 P 1fd4acc8159e428bb11f17af176c3542 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:21:53.562317 12349 raft_consensus.cc:686] T 389759e9386e478a8bda705033c55cf9 P 1fd4acc8159e428bb11f17af176c3542 [term 1 LEADER]: Becoming Leader. State: Replica: 1fd4acc8159e428bb11f17af176c3542, State: Running, Role: LEADER
I20260808 22:21:53.562455 12349 consensus_queue.cc:237] T 389759e9386e478a8bda705033c55cf9 P 1fd4acc8159e428bb11f17af176c3542 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9d58c0fe34448c8963c0ca696b3c1b" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44775 } } peers { permanent_uuid: "50ce356aabd9458197537fddca193355" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 46851 } } peers { permanent_uuid: "1fd4acc8159e428bb11f17af176c3542" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46643 } }
I20260808 22:21:53.563124 12325 tablet_replica.cc:439] T 389759e9386e478a8bda705033c55cf9 P 1fd4acc8159e428bb11f17af176c3542: TxnStatusTablet state changed. Reason: New leader 1fd4acc8159e428bb11f17af176c3542. Latest consensus state: current_term: 1 leader_uuid: "1fd4acc8159e428bb11f17af176c3542" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9d58c0fe34448c8963c0ca696b3c1b" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44775 } } peers { permanent_uuid: "50ce356aabd9458197537fddca193355" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 46851 } } peers { permanent_uuid: "1fd4acc8159e428bb11f17af176c3542" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46643 } } }
I20260808 22:21:53.563243 12325 tablet_replica.cc:442] T 389759e9386e478a8bda705033c55cf9 P 1fd4acc8159e428bb11f17af176c3542: This TxnStatusTablet replica's current role is: LEADER
I20260808 22:21:53.563582 12380 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260808 22:21:53.563805 12380 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260808 22:21:53.564190 11590 catalog_manager.cc:5495] T 389759e9386e478a8bda705033c55cf9 P 1fd4acc8159e428bb11f17af176c3542 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1fd4acc8159e428bb11f17af176c3542 (127.9.176.195). New cstate: current_term: 1 leader_uuid: "1fd4acc8159e428bb11f17af176c3542" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9d58c0fe34448c8963c0ca696b3c1b" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50ce356aabd9458197537fddca193355" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 46851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fd4acc8159e428bb11f17af176c3542" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46643 } health_report { overall_health: HEALTHY } } }
I20260808 22:21:53.598021 12373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:53.594846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.195:46643 (local address 127.0.0.1:58090)
0808 22:21:53.595006 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0808 22:21:53.595021 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 22:21:53.595140 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:21:53.595414 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:21:53.595417 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:21:53.595424 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 22:21:53.595522 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:21:53.595527 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:21:53.596235 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:21:53.596239 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:21:53.596837 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:21:53.596845 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:21:53.597304 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:21:53.597308 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:21:53.597391 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:53.597436 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 22:21:53.597735 (+ 299us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 22:21:53.597750 (+ 15us) client_negotiation.cc:769] Sending connection context
0808 22:21:53.597808 (+ 58us) client_negotiation.cc:240] Negotiation successful
0808 22:21:53.597891 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260808 22:21:53.598021 12312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:53.594972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58090 (local address 127.9.176.195:46643)
0808 22:21:53.595108 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0808 22:21:53.595113 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:21:53.595127 (+ 14us) server_negotiation.cc:407] Connection header received
0808 22:21:53.595266 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:21:53.595270 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:21:53.595318 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:21:53.595377 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:21:53.595633 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:53.596125 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:53.596981 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:53.597200 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:53.597288 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:53.597569 (+ 281us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:21:53.597651 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:21:53.597712 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0808 22:21:53.597831 (+ 119us) server_negotiation.cc:299] Negotiation successful
0808 22:21:53.597882 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 22:21:53.600710 12031 raft_consensus.cc:1261] T 389759e9386e478a8bda705033c55cf9 P 50ce356aabd9458197537fddca193355 [term 1 FOLLOWER]: Refusing update from remote peer 1fd4acc8159e428bb11f17af176c3542: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:21:53.600768 11811 raft_consensus.cc:1261] T 389759e9386e478a8bda705033c55cf9 P bf9d58c0fe34448c8963c0ca696b3c1b [term 1 FOLLOWER]: Refusing update from remote peer 1fd4acc8159e428bb11f17af176c3542: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:21:53.601150 12349 consensus_queue.cc:1035] T 389759e9386e478a8bda705033c55cf9 P 1fd4acc8159e428bb11f17af176c3542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50ce356aabd9458197537fddca193355" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 46851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:21:53.601305 12325 consensus_queue.cc:1035] T 389759e9386e478a8bda705033c55cf9 P 1fd4acc8159e428bb11f17af176c3542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf9d58c0fe34448c8963c0ca696b3c1b" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:21:53.603050 12330 tablet_replica.cc:439] T 389759e9386e478a8bda705033c55cf9 P 50ce356aabd9458197537fddca193355: TxnStatusTablet state changed. Reason: New leader 1fd4acc8159e428bb11f17af176c3542. Latest consensus state: current_term: 1 leader_uuid: "1fd4acc8159e428bb11f17af176c3542" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9d58c0fe34448c8963c0ca696b3c1b" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44775 } } peers { permanent_uuid: "50ce356aabd9458197537fddca193355" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 46851 } } peers { permanent_uuid: "1fd4acc8159e428bb11f17af176c3542" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46643 } } }
I20260808 22:21:53.603164 12330 tablet_replica.cc:442] T 389759e9386e478a8bda705033c55cf9 P 50ce356aabd9458197537fddca193355: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 22:21:53.604236 12329 tablet_replica.cc:439] T 389759e9386e478a8bda705033c55cf9 P bf9d58c0fe34448c8963c0ca696b3c1b: TxnStatusTablet state changed. Reason: New leader 1fd4acc8159e428bb11f17af176c3542. Latest consensus state: current_term: 1 leader_uuid: "1fd4acc8159e428bb11f17af176c3542" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9d58c0fe34448c8963c0ca696b3c1b" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44775 } } peers { permanent_uuid: "50ce356aabd9458197537fddca193355" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 46851 } } peers { permanent_uuid: "1fd4acc8159e428bb11f17af176c3542" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46643 } } }
I20260808 22:21:53.604360 12329 tablet_replica.cc:442] T 389759e9386e478a8bda705033c55cf9 P bf9d58c0fe34448c8963c0ca696b3c1b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 22:21:53.605371 12325 tablet_replica.cc:439] T 389759e9386e478a8bda705033c55cf9 P 1fd4acc8159e428bb11f17af176c3542: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1fd4acc8159e428bb11f17af176c3542" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9d58c0fe34448c8963c0ca696b3c1b" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44775 } } peers { permanent_uuid: "50ce356aabd9458197537fddca193355" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 46851 } } peers { permanent_uuid: "1fd4acc8159e428bb11f17af176c3542" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46643 } } }
I20260808 22:21:53.605479 12325 tablet_replica.cc:442] T 389759e9386e478a8bda705033c55cf9 P 1fd4acc8159e428bb11f17af176c3542: This TxnStatusTablet replica's current role is: LEADER
I20260808 22:21:53.606312 12336 tablet_replica.cc:439] T 389759e9386e478a8bda705033c55cf9 P 1fd4acc8159e428bb11f17af176c3542: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1fd4acc8159e428bb11f17af176c3542" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9d58c0fe34448c8963c0ca696b3c1b" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44775 } } peers { permanent_uuid: "50ce356aabd9458197537fddca193355" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 46851 } } peers { permanent_uuid: "1fd4acc8159e428bb11f17af176c3542" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46643 } } }
I20260808 22:21:53.606436 12336 tablet_replica.cc:442] T 389759e9386e478a8bda705033c55cf9 P 1fd4acc8159e428bb11f17af176c3542: This TxnStatusTablet replica's current role is: LEADER
I20260808 22:21:53.607486 12329 tablet_replica.cc:439] T 389759e9386e478a8bda705033c55cf9 P bf9d58c0fe34448c8963c0ca696b3c1b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1fd4acc8159e428bb11f17af176c3542" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9d58c0fe34448c8963c0ca696b3c1b" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44775 } } peers { permanent_uuid: "50ce356aabd9458197537fddca193355" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 46851 } } peers { permanent_uuid: "1fd4acc8159e428bb11f17af176c3542" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46643 } } }
I20260808 22:21:53.607584 12329 tablet_replica.cc:442] T 389759e9386e478a8bda705033c55cf9 P bf9d58c0fe34448c8963c0ca696b3c1b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 22:21:53.607781 12330 tablet_replica.cc:439] T 389759e9386e478a8bda705033c55cf9 P 50ce356aabd9458197537fddca193355: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1fd4acc8159e428bb11f17af176c3542" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9d58c0fe34448c8963c0ca696b3c1b" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44775 } } peers { permanent_uuid: "50ce356aabd9458197537fddca193355" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 46851 } } peers { permanent_uuid: "1fd4acc8159e428bb11f17af176c3542" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46643 } } }
I20260808 22:21:53.607892 12330 tablet_replica.cc:442] T 389759e9386e478a8bda705033c55cf9 P 50ce356aabd9458197537fddca193355: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 22:21:53.616652 12317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:53.612995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36800 (local address 127.9.176.193:44775)
0808 22:21:53.613354 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0808 22:21:53.613357 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:21:53.613367 (+ 10us) server_negotiation.cc:407] Connection header received
0808 22:21:53.613425 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:21:53.613427 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:21:53.613479 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:21:53.613577 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:21:53.613924 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:53.614500 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:53.615309 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:53.615519 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:53.615603 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:53.615863 (+ 260us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:21:53.616296 (+ 433us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:21:53.616398 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0808 22:21:53.616477 (+ 79us) server_negotiation.cc:299] Negotiation successful
0808 22:21:53.616529 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293}
I20260808 22:21:53.627379 12386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:53.623371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.195:46643 (local address 127.9.176.193:60643)
0808 22:21:53.623650 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0808 22:21:53.623663 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 22:21:53.623745 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:21:53.623965 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:21:53.623967 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:21:53.623972 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 22:21:53.624063 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:21:53.624067 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:21:53.625004 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:21:53.625007 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:21:53.626186 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:21:53.626196 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:21:53.626757 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:21:53.626760 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:21:53.626838 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:53.626881 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 22:21:53.627151 (+ 270us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 22:21:53.627155 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 22:21:53.627197 (+ 42us) client_negotiation.cc:240] Negotiation successful
0808 22:21:53.627257 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":109,"threads_started":1}
I20260808 22:21:53.627383 12312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:53.623487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.193:60643 (local address 127.9.176.195:46643)
0808 22:21:53.623622 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0808 22:21:53.623626 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:21:53.623724 (+ 98us) server_negotiation.cc:407] Connection header received
0808 22:21:53.623825 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:21:53.623829 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:21:53.623877 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:21:53.623955 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:21:53.624142 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:53.624878 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:53.626296 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:53.626641 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:53.626739 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:53.626994 (+ 255us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:21:53.627079 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:21:53.627147 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0808 22:21:53.627210 (+ 63us) server_negotiation.cc:299] Negotiation successful
0808 22:21:53.627257 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 22:21:53.634274 12031 raft_consensus.cc:1261] T c8e16c9346ed4f1b838d65d6b985ea60 P 50ce356aabd9458197537fddca193355 [term 1 FOLLOWER]: Refusing update from remote peer bf9d58c0fe34448c8963c0ca696b3c1b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:21:53.634320 12220 raft_consensus.cc:1261] T c8e16c9346ed4f1b838d65d6b985ea60 P 1fd4acc8159e428bb11f17af176c3542 [term 1 FOLLOWER]: Refusing update from remote peer bf9d58c0fe34448c8963c0ca696b3c1b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:21:53.634759 12329 consensus_queue.cc:1035] T c8e16c9346ed4f1b838d65d6b985ea60 P bf9d58c0fe34448c8963c0ca696b3c1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "50ce356aabd9458197537fddca193355" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 46851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:21:53.635093 12329 consensus_queue.cc:1035] T c8e16c9346ed4f1b838d65d6b985ea60 P bf9d58c0fe34448c8963c0ca696b3c1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fd4acc8159e428bb11f17af176c3542" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:21:53.655303 12293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:53.652133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.193:44775 (local address 127.9.176.195:60177)
0808 22:21:53.652277 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0808 22:21:53.652294 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 22:21:53.652376 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:21:53.652574 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:21:53.652576 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:21:53.652585 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 22:21:53.652752 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:21:53.652758 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:21:53.653491 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:21:53.653495 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:21:53.654094 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:21:53.654101 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:21:53.654607 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:21:53.654611 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:21:53.654715 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:53.654769 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 22:21:53.655037 (+ 268us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 22:21:53.655044 (+ 7us) client_negotiation.cc:769] Sending connection context
0808 22:21:53.655104 (+ 60us) client_negotiation.cc:240] Negotiation successful
0808 22:21:53.655198 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260808 22:21:53.655299 12317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:53.652184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.195:60177 (local address 127.9.176.193:44775)
0808 22:21:53.652324 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0808 22:21:53.652328 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:21:53.652341 (+ 13us) server_negotiation.cc:407] Connection header received
0808 22:21:53.652454 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:21:53.652457 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:21:53.652500 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:21:53.652559 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:21:53.652855 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:53.653395 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:53.654211 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:53.654492 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:53.654590 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:53.654894 (+ 304us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:21:53.654958 (+ 64us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:21:53.655011 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0808 22:21:53.655124 (+ 113us) server_negotiation.cc:299] Negotiation successful
0808 22:21:53.655171 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260808 22:21:53.679239 11590 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60042:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:21:53.679392 11590 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60042:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:21:53.681416 11590 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4533bca6608847efb2cf76f437edff0c: Sending DeleteTablet for 3 replicas of tablet c8e16c9346ed4f1b838d65d6b985ea60
I20260808 22:21:53.682025 11785 tablet_service.cc:1514] Processing DeleteTablet for tablet c8e16c9346ed4f1b838d65d6b985ea60 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:21:53 UTC) from {username='kudu', principal='kudu/127.9.176.254@KRBTEST.COM'} at 127.0.0.1:36790
I20260808 22:21:53.682058 12198 tablet_service.cc:1514] Processing DeleteTablet for tablet c8e16c9346ed4f1b838d65d6b985ea60 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:21:53 UTC) from {username='kudu', principal='kudu/127.9.176.254@KRBTEST.COM'} at 127.0.0.1:58068
I20260808 22:21:53.682228 12355 tablet_replica.cc:330] T c8e16c9346ed4f1b838d65d6b985ea60 P 1fd4acc8159e428bb11f17af176c3542: stopping tablet replica
I20260808 22:21:53.682229 12356 tablet_replica.cc:330] T c8e16c9346ed4f1b838d65d6b985ea60 P bf9d58c0fe34448c8963c0ca696b3c1b: stopping tablet replica
I20260808 22:21:53.682330 12356 raft_consensus.cc:2229] T c8e16c9346ed4f1b838d65d6b985ea60 P bf9d58c0fe34448c8963c0ca696b3c1b [term 1 LEADER]: Raft consensus shutting down.
I20260808 22:21:53.682333 12355 raft_consensus.cc:2229] T c8e16c9346ed4f1b838d65d6b985ea60 P 1fd4acc8159e428bb11f17af176c3542 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:21:53.682426 12355 raft_consensus.cc:2258] T c8e16c9346ed4f1b838d65d6b985ea60 P 1fd4acc8159e428bb11f17af176c3542 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:21:53.682595 12356 raft_consensus.cc:2258] T c8e16c9346ed4f1b838d65d6b985ea60 P bf9d58c0fe34448c8963c0ca696b3c1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:21:53.683382 12355 ts_tablet_manager.cc:1905] T c8e16c9346ed4f1b838d65d6b985ea60 P 1fd4acc8159e428bb11f17af176c3542: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:21:53.683516 12002 tablet_service.cc:1514] Processing DeleteTablet for tablet c8e16c9346ed4f1b838d65d6b985ea60 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:21:53 UTC) from {username='kudu', principal='kudu/127.9.176.254@KRBTEST.COM'} at 127.0.0.1:40638
I20260808 22:21:53.684749 12357 tablet_replica.cc:330] T c8e16c9346ed4f1b838d65d6b985ea60 P 50ce356aabd9458197537fddca193355: stopping tablet replica
I20260808 22:21:53.684868 12357 raft_consensus.cc:2229] T c8e16c9346ed4f1b838d65d6b985ea60 P 50ce356aabd9458197537fddca193355 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:21:53.685006 12357 raft_consensus.cc:2258] T c8e16c9346ed4f1b838d65d6b985ea60 P 50ce356aabd9458197537fddca193355 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:21:53.685387 12356 ts_tablet_manager.cc:1905] T c8e16c9346ed4f1b838d65d6b985ea60 P bf9d58c0fe34448c8963c0ca696b3c1b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:21:53.686967 12355 ts_tablet_manager.cc:1918] T c8e16c9346ed4f1b838d65d6b985ea60 P 1fd4acc8159e428bb11f17af176c3542: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 22:21:53.687036 12355 log.cc:1196] T c8e16c9346ed4f1b838d65d6b985ea60 P 1fd4acc8159e428bb11f17af176c3542: Deleting WAL directory at /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/minicluster-data/ts-2/wal/wals/c8e16c9346ed4f1b838d65d6b985ea60
I20260808 22:21:53.687268 12355 ts_tablet_manager.cc:1939] T c8e16c9346ed4f1b838d65d6b985ea60 P 1fd4acc8159e428bb11f17af176c3542: Deleting consensus metadata
I20260808 22:21:53.687947 11568 catalog_manager.cc:4841] TS 1fd4acc8159e428bb11f17af176c3542 (127.9.176.195:46643): tablet c8e16c9346ed4f1b838d65d6b985ea60 (table test-table [id=f97096aa2fe44efebe627eb5acd27401]) successfully deleted
I20260808 22:21:53.688333 12356 ts_tablet_manager.cc:1918] T c8e16c9346ed4f1b838d65d6b985ea60 P bf9d58c0fe34448c8963c0ca696b3c1b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 22:21:53.688400 12356 log.cc:1196] T c8e16c9346ed4f1b838d65d6b985ea60 P bf9d58c0fe34448c8963c0ca696b3c1b: Deleting WAL directory at /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/minicluster-data/ts-0/wal/wals/c8e16c9346ed4f1b838d65d6b985ea60
I20260808 22:21:53.688637 12356 ts_tablet_manager.cc:1939] T c8e16c9346ed4f1b838d65d6b985ea60 P bf9d58c0fe34448c8963c0ca696b3c1b: Deleting consensus metadata
I20260808 22:21:53.688614 12357 ts_tablet_manager.cc:1905] T c8e16c9346ed4f1b838d65d6b985ea60 P 50ce356aabd9458197537fddca193355: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:21:53.689976 11572 catalog_manager.cc:4841] TS bf9d58c0fe34448c8963c0ca696b3c1b (127.9.176.193:44775): tablet c8e16c9346ed4f1b838d65d6b985ea60 (table test-table [id=f97096aa2fe44efebe627eb5acd27401]) successfully deleted
I20260808 22:21:53.691223 12357 ts_tablet_manager.cc:1918] T c8e16c9346ed4f1b838d65d6b985ea60 P 50ce356aabd9458197537fddca193355: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 22:21:53.691306 12357 log.cc:1196] T c8e16c9346ed4f1b838d65d6b985ea60 P 50ce356aabd9458197537fddca193355: Deleting WAL directory at /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786227706282457-9923-0/minicluster-data/ts-1/wal/wals/c8e16c9346ed4f1b838d65d6b985ea60
I20260808 22:21:53.691548 12357 ts_tablet_manager.cc:1939] T c8e16c9346ed4f1b838d65d6b985ea60 P 50ce356aabd9458197537fddca193355: Deleting consensus metadata
I20260808 22:21:53.692191 11567 catalog_manager.cc:4841] TS 50ce356aabd9458197537fddca193355 (127.9.176.194:46851): tablet c8e16c9346ed4f1b838d65d6b985ea60 (table test-table [id=f97096aa2fe44efebe627eb5acd27401]) successfully deleted
Aug 08 22:21:53 dist-test-slave-ncp9 krb5kdc[11523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227713, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.176.193@KRBTEST.COM
I20260808 22:21:53.701014 12317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:53.690558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36810 (local address 127.9.176.193:44775)
0808 22:21:53.690666 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0808 22:21:53.690670 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:21:53.691232 (+ 562us) server_negotiation.cc:407] Connection header received
0808 22:21:53.691570 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:21:53.691574 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:21:53.691623 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:21:53.691678 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:21:53.692722 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:53.693297 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:53.694121 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:53.694339 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:53.694424 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:53.697372 (+ 2948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:21:53.697383 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:21:53.697400 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:21:53.697423 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:21:53.699044 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:21:53.699458 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:21:53.699464 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:21:53.699465 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:21:53.699542 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:21:53.699805 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:21:53.699808 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:21:53.699810 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:21:53.700136 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:21:53.700340 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0808 22:21:53.700538 (+ 198us) server_negotiation.cc:299] Negotiation successful
0808 22:21:53.700653 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
W20260808 22:21:53.701545 11862 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36810
I20260808 22:21:53.710790 11657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:53.703184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60060 (local address 127.9.176.254:33915)
0808 22:21:53.703316 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0808 22:21:53.703320 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:21:53.703689 (+ 369us) server_negotiation.cc:407] Connection header received
0808 22:21:53.703791 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:21:53.703794 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:21:53.703843 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:21:53.703906 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:21:53.704528 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:53.705200 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:53.706132 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:53.706406 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:53.706495 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:53.707844 (+ 1349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:21:53.707848 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:21:53.707850 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:21:53.707876 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:21:53.709231 (+ 1355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:21:53.709608 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:21:53.709612 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:21:53.709614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:21:53.709739 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:21:53.709985 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:21:53.709988 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:21:53.709990 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:21:53.710170 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:21:53.710293 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0808 22:21:53.710511 (+ 218us) server_negotiation.cc:299] Negotiation successful
0808 22:21:53.710632 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260808 22:21:53.711150 11590 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60060
I20260808 22:21:53.712433 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 11688
I20260808 22:21:53.724429 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 11903
I20260808 22:21:53.734980 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 12106
I20260808 22:21:53.747416 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 11542
2026-08-08T22:21:53Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2664 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 22:21:53 dist-test-slave-ncp9 krb5kdc[12408](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 22:21:53 dist-test-slave-ncp9 krb5kdc[12408](info): set up 2 sockets
Aug 08 22:21:53 dist-test-slave-ncp9 krb5kdc[12408](info): commencing operation
krb5kdc: starting...
W20260808 22:21:54.866909 9923 mini_kdc.cc:121] Time spent starting KDC: real 1.095s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/krb5kdc/test-user.keytab.
Aug 08 22:21:54 dist-test-slave-ncp9 krb5kdc[12408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T22:21:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:21:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.176.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.176.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.176.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab.
Entry for principal kudu/127.9.176.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab.
I20260808 22:21:54.984413 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:44743
--webserver_interface=127.9.176.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab
--principal=kudu/127.9.176.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:45665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.176.254:44743
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:21:55.091590 12425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:21:55.091863 12425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:21:55.091918 12425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:21:55.095837 12425 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:21:55.095897 12425 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260808 22:21:55.095920 12425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:21:55.095942 12425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:21:55.095965 12425 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:21:55.095992 12425 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:21:55.101176 12425 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:45665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.176.254:44743
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab
--principal=kudu/127.9.176.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:44743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.176.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12425
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:21:55.102233 12425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:21:55.103257 12425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:21:55.111539 12431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:21:55.111622 12434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:21:55.111538 12436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:21:55.111744 12425 server_base.cc:1034] running on GCE node
I20260808 22:21:55.112332 12425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:21:55.113431 12425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:21:55.114598 12425 hybrid_clock.cc:648] HybridClock initialized: now 1786227715114579 us; error 30 us; skew 500 ppm
Aug 08 22:21:55 dist-test-slave-ncp9 krb5kdc[12408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227715, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:21:55.117228 12425 init.cc:377] Logged in from keytab as kudu/127.9.176.254@KRBTEST.COM (short username kudu)
I20260808 22:21:55.118115 12425 webserver.cc:460] Webserver started at http://127.9.176.254:34947/ using document root <none> and password file <none>
I20260808 22:21:55.118628 12425 fs_manager.cc:359] Metadata directory not provided
I20260808 22:21:55.118697 12425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:21:55.118902 12425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:21:55.120869 12425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/minicluster-data/master-0/data/instance:
uuid: "a9a81a719cd649ec800f8a3cc702574d"
format_stamp: "Formatted at 2026-08-08 22:21:55 on dist-test-slave-ncp9"
server_key: "814604d9b5fde310ae2d10a969a7bd6a"
server_key_iv: "22896b80e9b73531eee332e7af99f486"
server_key_version: "encryptionkey@0"
I20260808 22:21:55.121515 12425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/minicluster-data/master-0/wal/instance:
uuid: "a9a81a719cd649ec800f8a3cc702574d"
format_stamp: "Formatted at 2026-08-08 22:21:55 on dist-test-slave-ncp9"
server_key: "814604d9b5fde310ae2d10a969a7bd6a"
server_key_iv: "22896b80e9b73531eee332e7af99f486"
server_key_version: "encryptionkey@0"
I20260808 22:21:55.125030 12425 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260808 22:21:55.127223 12445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:21:55.128365 12425 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 22:21:55.128494 12425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/minicluster-data/master-0/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/minicluster-data/master-0/wal
uuid: "a9a81a719cd649ec800f8a3cc702574d"
format_stamp: "Formatted at 2026-08-08 22:21:55 on dist-test-slave-ncp9"
server_key: "814604d9b5fde310ae2d10a969a7bd6a"
server_key_iv: "22896b80e9b73531eee332e7af99f486"
server_key_version: "encryptionkey@0"
I20260808 22:21:55.128587 12425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:21:55.145262 12425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:21:55.147745 12425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:21:55.147913 12425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:21:55.155309 12425 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.254:44743
I20260808 22:21:55.155329 12545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.254:44743 every 8 connection(s)
I20260808 22:21:55.156185 12425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
I20260808 22:21:55.158910 12546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:21:55.160041 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 12425
I20260808 22:21:55.160157 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/minicluster-data/master-0/wal/instance
I20260808 22:21:55.160451 9923 external_mini_cluster.cc:1428] Setting key ab6c2ef39fd7c93a84073a83438d9740
I20260808 22:21:55.165131 12546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9a81a719cd649ec800f8a3cc702574d: Bootstrap starting.
I20260808 22:21:55.167630 12546 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a9a81a719cd649ec800f8a3cc702574d: Neither blocks nor log segments found. Creating new log.
I20260808 22:21:55.168530 12546 log.cc:824] T 00000000000000000000000000000000 P a9a81a719cd649ec800f8a3cc702574d: Log is configured to *not* fsync() on all Append() calls
Aug 08 22:21:55 dist-test-slave-ncp9 krb5kdc[12408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
I20260808 22:21:55.170535 12546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9a81a719cd649ec800f8a3cc702574d: No bootstrap required, opened a new log
I20260808 22:21:55.174196 12549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:55.161811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40608 (local address 127.9.176.254:44743)
0808 22:21:55.162201 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0808 22:21:55.162216 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 22:21:55.162251 (+ 35us) server_negotiation.cc:407] Connection header received
0808 22:21:55.162961 (+ 710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:21:55.162992 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:21:55.163311 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:21:55.163579 (+ 268us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:21:55.164422 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:55.165479 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:55.166614 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:55.167064 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:55.167174 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:55.169868 (+ 2694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:21:55.169873 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:21:55.169886 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:21:55.169912 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:21:55.171520 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:21:55.172005 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:21:55.172009 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:21:55.172013 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:21:55.172117 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:21:55.172378 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:21:55.172380 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:21:55.172381 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:21:55.172735 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:21:55.172915 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0808 22:21:55.173198 (+ 283us) server_negotiation.cc:299] Negotiation successful
0808 22:21:55.173471 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":104,"threads_started":1}
I20260808 22:21:55.174372 12546 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9a81a719cd649ec800f8a3cc702574d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9a81a719cd649ec800f8a3cc702574d" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 44743 } }
I20260808 22:21:55.174593 12546 raft_consensus.cc:374] T 00000000000000000000000000000000 P a9a81a719cd649ec800f8a3cc702574d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:21:55.174659 12546 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9a81a719cd649ec800f8a3cc702574d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9a81a719cd649ec800f8a3cc702574d, State: Initialized, Role: FOLLOWER
I20260808 22:21:55.175074 12546 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9a81a719cd649ec800f8a3cc702574d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9a81a719cd649ec800f8a3cc702574d" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 44743 } }
I20260808 22:21:55.175213 12546 raft_consensus.cc:388] T 00000000000000000000000000000000 P a9a81a719cd649ec800f8a3cc702574d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:21:55.175279 12546 raft_consensus.cc:482] T 00000000000000000000000000000000 P a9a81a719cd649ec800f8a3cc702574d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:21:55.175370 12546 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9a81a719cd649ec800f8a3cc702574d [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:21:55.176415 12546 raft_consensus.cc:504] T 00000000000000000000000000000000 P a9a81a719cd649ec800f8a3cc702574d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9a81a719cd649ec800f8a3cc702574d" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 44743 } }
I20260808 22:21:55.176750 12546 leader_election.cc:302] T 00000000000000000000000000000000 P a9a81a719cd649ec800f8a3cc702574d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9a81a719cd649ec800f8a3cc702574d; no voters:
I20260808 22:21:55.177059 12546 leader_election.cc:288] T 00000000000000000000000000000000 P a9a81a719cd649ec800f8a3cc702574d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:21:55.177135 12551 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a9a81a719cd649ec800f8a3cc702574d [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:21:55.177337 12551 raft_consensus.cc:686] T 00000000000000000000000000000000 P a9a81a719cd649ec800f8a3cc702574d [term 1 LEADER]: Becoming Leader. State: Replica: a9a81a719cd649ec800f8a3cc702574d, State: Running, Role: LEADER
I20260808 22:21:55.177656 12551 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9a81a719cd649ec800f8a3cc702574d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9a81a719cd649ec800f8a3cc702574d" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 44743 } }
I20260808 22:21:55.178189 12546 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9a81a719cd649ec800f8a3cc702574d [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:21:55.179111 12555 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9a81a719cd649ec800f8a3cc702574d [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9a81a719cd649ec800f8a3cc702574d. Latest consensus state: current_term: 1 leader_uuid: "a9a81a719cd649ec800f8a3cc702574d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9a81a719cd649ec800f8a3cc702574d" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 44743 } } }
I20260808 22:21:55.179236 12555 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9a81a719cd649ec800f8a3cc702574d [sys.catalog]: This master's current role is: LEADER
I20260808 22:21:55.179651 12564 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:21:55.179683 12552 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9a81a719cd649ec800f8a3cc702574d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a9a81a719cd649ec800f8a3cc702574d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9a81a719cd649ec800f8a3cc702574d" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 44743 } } }
I20260808 22:21:55.179793 12552 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9a81a719cd649ec800f8a3cc702574d [sys.catalog]: This master's current role is: LEADER
I20260808 22:21:55.182255 12564 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:21:55.187369 12564 catalog_manager.cc:1347] Generated new cluster ID: 64159994bc024890aad42f203b93819e
I20260808 22:21:55.187451 12564 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:21:55.209359 12564 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:21:55.210350 12564 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:21:55.216481 12564 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a9a81a719cd649ec800f8a3cc702574d: Generated new TSK 0
I20260808 22:21:55.217180 12564 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.176.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.176.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.176.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.193.keytab.
Entry for principal kudu/127.9.176.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.193.keytab.
I20260808 22:21:55.251389 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.193:0
--local_ip_for_outbound_sockets=127.9.176.193
--webserver_interface=127.9.176.193
--webserver_port=0
--tserver_master_addrs=127.9.176.254:44743
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.193.keytab
--principal=kudu/127.9.176.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:45665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:21:55.358954 12585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:21:55.359189 12585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:21:55.359236 12585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:21:55.363220 12585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:21:55.363270 12585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:21:55.363317 12585 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260808 22:21:55.363345 12585 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 22:21:55.363384 12585 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.193
I20260808 22:21:55.368498 12585 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:45665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.193.keytab
--principal=kudu/127.9.176.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.176.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.176.254:44743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12585
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.193
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:21:55.369489 12585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:21:55.370424 12585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:21:55.379293 12596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:21:55.379307 12594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:21:55.379305 12593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:21:55.379689 12585 server_base.cc:1034] running on GCE node
I20260808 22:21:55.380069 12585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:21:55.380628 12585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:21:55.381778 12585 hybrid_clock.cc:648] HybridClock initialized: now 1786227715381761 us; error 28 us; skew 500 ppm
Aug 08 22:21:55 dist-test-slave-ncp9 krb5kdc[12408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227715, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:21:55.384262 12585 init.cc:377] Logged in from keytab as kudu/127.9.176.193@KRBTEST.COM (short username kudu)
I20260808 22:21:55.385082 12585 webserver.cc:460] Webserver started at http://127.9.176.193:40737/ using document root <none> and password file <none>
I20260808 22:21:55.385581 12585 fs_manager.cc:359] Metadata directory not provided
I20260808 22:21:55.385643 12585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:21:55.385820 12585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:21:55.387717 12585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/minicluster-data/ts-0/data/instance:
uuid: "ec50da59889f4a1e84a835ff268b75e8"
format_stamp: "Formatted at 2026-08-08 22:21:55 on dist-test-slave-ncp9"
server_key: "d77077dc4eda61d2b750ee1dfb064eb4"
server_key_iv: "fc2bd4cd006ca04e4a77913861e6c6ad"
server_key_version: "encryptionkey@0"
I20260808 22:21:55.388224 12585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/minicluster-data/ts-0/wal/instance:
uuid: "ec50da59889f4a1e84a835ff268b75e8"
format_stamp: "Formatted at 2026-08-08 22:21:55 on dist-test-slave-ncp9"
server_key: "d77077dc4eda61d2b750ee1dfb064eb4"
server_key_iv: "fc2bd4cd006ca04e4a77913861e6c6ad"
server_key_version: "encryptionkey@0"
I20260808 22:21:55.391613 12585 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260808 22:21:55.393913 12608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:21:55.395030 12585 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 22:21:55.395131 12585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/minicluster-data/ts-0/wal
uuid: "ec50da59889f4a1e84a835ff268b75e8"
format_stamp: "Formatted at 2026-08-08 22:21:55 on dist-test-slave-ncp9"
server_key: "d77077dc4eda61d2b750ee1dfb064eb4"
server_key_iv: "fc2bd4cd006ca04e4a77913861e6c6ad"
server_key_version: "encryptionkey@0"
I20260808 22:21:55.395219 12585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:21:55.404271 12585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:21:55.406718 12585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:21:55.406874 12585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:21:55.408159 12585 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:21:55.408205 12585 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:21:55.408254 12585 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:21:55.408293 12585 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 22:21:55 dist-test-slave-ncp9 krb5kdc[12408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227715, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.193@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
I20260808 22:21:55.418045 12585 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.193:34469
I20260808 22:21:55.418118 12784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.193:34469 every 8 connection(s)
I20260808 22:21:55.419027 12585 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
I20260808 22:21:55.421180 12549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:55.408884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.193:60085 (local address 127.9.176.254:44743)
0808 22:21:55.409016 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0808 22:21:55.409019 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:21:55.409962 (+ 943us) server_negotiation.cc:407] Connection header received
0808 22:21:55.410958 (+ 996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:21:55.410962 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:21:55.411019 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:21:55.411073 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:21:55.412680 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:55.413360 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:55.414512 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:55.414801 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:55.414896 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:55.417821 (+ 2925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:21:55.417825 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:21:55.417828 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:21:55.417859 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:21:55.419234 (+ 1375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:21:55.419896 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:21:55.419899 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:21:55.419900 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:21:55.419992 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:21:55.420300 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:21:55.420302 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:21:55.420303 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:21:55.420460 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:21:55.420531 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0808 22:21:55.420951 (+ 420us) server_negotiation.cc:299] Negotiation successful
0808 22:21:55.421063 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260808 22:21:55.421890 12618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:55.409381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:44743 (local address 127.9.176.193:60085)
0808 22:21:55.409825 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0808 22:21:55.409864 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 22:21:55.410729 (+ 865us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:21:55.411474 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:21:55.411483 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:21:55.411913 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:21:55.412482 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:21:55.412503 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:21:55.413494 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:21:55.413497 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:21:55.414369 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:21:55.414379 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:21:55.414974 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:21:55.414979 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:21:55.415136 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:55.416180 (+ 1044us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:21:55.416201 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:21:55.417653 (+ 1452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:21:55.419338 (+ 1685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:21:55.419344 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:21:55.419357 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:21:55.419677 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:21:55.420074 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:21:55.420077 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:21:55.420078 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:21:55.420225 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:21:55.420531 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:21:55.420536 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:21:55.420745 (+ 209us) client_negotiation.cc:769] Sending connection context
0808 22:21:55.420914 (+ 169us) client_negotiation.cc:240] Negotiation successful
0808 22:21:55.421180 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":118,"threads_started":1}
I20260808 22:21:55.423101 12786 heartbeater.cc:347] Connected to a master server at 127.9.176.254:44743
I20260808 22:21:55.423234 12786 heartbeater.cc:464] Registering TS with master...
I20260808 22:21:55.423717 12786 heartbeater.cc:511] Master 127.9.176.254:44743 requested a full tablet report, sending...
I20260808 22:21:55.425115 12464 ts_manager.cc:194] Registered new tserver with Master: ec50da59889f4a1e84a835ff268b75e8 (127.9.176.193:34469)
I20260808 22:21:55.426399 12464 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.176.193@KRBTEST.COM'} at 127.9.176.193:60085
I20260808 22:21:55.426898 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 12585
I20260808 22:21:55.427013 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/minicluster-data/ts-0/wal/instance
I20260808 22:21:55.427317 9923 external_mini_cluster.cc:1428] Setting key fd5a5df664f04bf89d7ac437d12c649e
I20260808 22:21:55.431701 12796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:55.424882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:44743 (local address 127.9.176.193:58691)
0808 22:21:55.425244 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0808 22:21:55.425258 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 22:21:55.425348 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:21:55.425552 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:21:55.425555 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:21:55.425748 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:21:55.425860 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:21:55.425868 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:21:55.426642 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:21:55.426645 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:21:55.427376 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:21:55.427386 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:21:55.427825 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:21:55.427829 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:21:55.427926 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:55.428389 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:21:55.428403 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:21:55.428809 (+ 406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:21:55.430383 (+ 1574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:21:55.430385 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:21:55.430387 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:21:55.430559 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:21:55.430829 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:21:55.430831 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:21:55.430833 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:21:55.430904 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:21:55.431236 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:21:55.431239 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:21:55.431316 (+ 77us) client_negotiation.cc:769] Sending connection context
0808 22:21:55.431398 (+ 82us) client_negotiation.cc:240] Negotiation successful
0808 22:21:55.431525 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"mutex_wait_us":1,"thread_start_us":84,"threads_started":1}
I20260808 22:21:55.431716 12549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:55.424984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.193:58691 (local address 127.9.176.254:44743)
0808 22:21:55.425097 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0808 22:21:55.425101 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:21:55.425316 (+ 215us) server_negotiation.cc:407] Connection header received
0808 22:21:55.425422 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:21:55.425424 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:21:55.425460 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:21:55.425518 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:21:55.425950 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:55.426475 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:55.427488 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:55.427707 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:55.427805 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:55.428947 (+ 1142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:21:55.428951 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:21:55.428953 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:21:55.428977 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:21:55.430280 (+ 1303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:21:55.430665 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:21:55.430669 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:21:55.430671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:21:55.430747 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:21:55.430983 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:21:55.430986 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:21:55.430988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:21:55.431148 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:21:55.431231 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0808 22:21:55.431483 (+ 252us) server_negotiation.cc:299] Negotiation successful
0808 22:21:55.431579 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":42}
WARNING: no policy specified for kudu/127.9.176.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.176.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.176.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.194.keytab.
Entry for principal kudu/127.9.176.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.194.keytab.
I20260808 22:21:55.455343 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.194:0
--local_ip_for_outbound_sockets=127.9.176.194
--webserver_interface=127.9.176.194
--webserver_port=0
--tserver_master_addrs=127.9.176.254:44743
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.194.keytab
--principal=kudu/127.9.176.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:45665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:21:55.562513 12803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:21:55.562759 12803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:21:55.562820 12803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:21:55.566841 12803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:21:55.566902 12803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:21:55.566955 12803 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260808 22:21:55.566980 12803 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 22:21:55.567025 12803 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.194
I20260808 22:21:55.572062 12803 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:45665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.194.keytab
--principal=kudu/127.9.176.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.176.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.176.254:44743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12803
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.194
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:21:55.573036 12803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:21:55.573976 12803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:21:55.583052 12817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:21:55.583086 12814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:21:55.583192 12803 server_base.cc:1034] running on GCE node
W20260808 22:21:55.583097 12810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:21:55.583667 12803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:21:55.584254 12803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:21:55.585400 12803 hybrid_clock.cc:648] HybridClock initialized: now 1786227715585370 us; error 43 us; skew 500 ppm
Aug 08 22:21:55 dist-test-slave-ncp9 krb5kdc[12408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227715, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:21:55.587985 12803 init.cc:377] Logged in from keytab as kudu/127.9.176.194@KRBTEST.COM (short username kudu)
I20260808 22:21:55.588783 12803 webserver.cc:460] Webserver started at http://127.9.176.194:40043/ using document root <none> and password file <none>
I20260808 22:21:55.589316 12803 fs_manager.cc:359] Metadata directory not provided
I20260808 22:21:55.589383 12803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:21:55.589563 12803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:21:55.591491 12803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/minicluster-data/ts-1/data/instance:
uuid: "bcba8d834ed44491a94b32067ba681ad"
format_stamp: "Formatted at 2026-08-08 22:21:55 on dist-test-slave-ncp9"
server_key: "868fcb06aaa7bbe380e3f1373b3a6534"
server_key_iv: "db8abc4a78f9033f8ddbb6f31fdf0ba4"
server_key_version: "encryptionkey@0"
I20260808 22:21:55.592026 12803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/minicluster-data/ts-1/wal/instance:
uuid: "bcba8d834ed44491a94b32067ba681ad"
format_stamp: "Formatted at 2026-08-08 22:21:55 on dist-test-slave-ncp9"
server_key: "868fcb06aaa7bbe380e3f1373b3a6534"
server_key_iv: "db8abc4a78f9033f8ddbb6f31fdf0ba4"
server_key_version: "encryptionkey@0"
I20260808 22:21:55.595469 12803 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260808 22:21:55.597663 12825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:21:55.598821 12803 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 22:21:55.598922 12803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/minicluster-data/ts-1/wal
uuid: "bcba8d834ed44491a94b32067ba681ad"
format_stamp: "Formatted at 2026-08-08 22:21:55 on dist-test-slave-ncp9"
server_key: "868fcb06aaa7bbe380e3f1373b3a6534"
server_key_iv: "db8abc4a78f9033f8ddbb6f31fdf0ba4"
server_key_version: "encryptionkey@0"
I20260808 22:21:55.599010 12803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:21:55.608505 12803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:21:55.611380 12803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:21:55.611541 12803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:21:55.612843 12803 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:21:55.612916 12803 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:21:55.612977 12803 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:21:55.613006 12803 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 22:21:55 dist-test-slave-ncp9 krb5kdc[12408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227715, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.194@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
I20260808 22:21:55.622822 12803 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.194:36839
I20260808 22:21:55.622910 12995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.194:36839 every 8 connection(s)
I20260808 22:21:55.623871 12803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
I20260808 22:21:55.626647 12549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:55.613608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.194:33205 (local address 127.9.176.254:44743)
0808 22:21:55.613722 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0808 22:21:55.613726 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:21:55.614692 (+ 966us) server_negotiation.cc:407] Connection header received
0808 22:21:55.615666 (+ 974us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:21:55.615670 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:21:55.615731 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:21:55.615791 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:21:55.617416 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:55.618246 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:55.619921 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:55.620258 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:55.620376 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:55.623208 (+ 2832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:21:55.623211 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:21:55.623214 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:21:55.623244 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:21:55.624661 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:21:55.625208 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:21:55.625211 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:21:55.625213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:21:55.625296 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:21:55.625627 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:21:55.625630 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:21:55.625632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:21:55.625807 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:21:55.625891 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0808 22:21:55.626371 (+ 480us) server_negotiation.cc:299] Negotiation successful
0808 22:21:55.626489 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260808 22:21:55.627400 12840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:55.614072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:44743 (local address 127.9.176.194:33205)
0808 22:21:55.614556 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0808 22:21:55.614594 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0808 22:21:55.615407 (+ 813us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:21:55.616195 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:21:55.616204 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:21:55.616598 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:21:55.617221 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:21:55.617242 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:21:55.618403 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:21:55.618407 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:21:55.619800 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:21:55.619811 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:21:55.620419 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:21:55.620423 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:21:55.620587 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:55.621192 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:21:55.621213 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:21:55.623020 (+ 1807us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:21:55.624777 (+ 1757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:21:55.624783 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:21:55.624796 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:21:55.625112 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:21:55.625378 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:21:55.625382 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:21:55.625384 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:21:55.625527 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:21:55.625906 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:21:55.625912 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:21:55.626138 (+ 226us) client_negotiation.cc:769] Sending connection context
0808 22:21:55.626289 (+ 151us) client_negotiation.cc:240] Negotiation successful
0808 22:21:55.626587 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":136,"threads_started":1}
I20260808 22:21:55.628464 12997 heartbeater.cc:347] Connected to a master server at 127.9.176.254:44743
I20260808 22:21:55.628620 12997 heartbeater.cc:464] Registering TS with master...
I20260808 22:21:55.629177 12997 heartbeater.cc:511] Master 127.9.176.254:44743 requested a full tablet report, sending...
I20260808 22:21:55.630198 12465 ts_manager.cc:194] Registered new tserver with Master: bcba8d834ed44491a94b32067ba681ad (127.9.176.194:36839)
I20260808 22:21:55.630748 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 12803
I20260808 22:21:55.630864 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/minicluster-data/ts-1/wal/instance
I20260808 22:21:55.630909 12465 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.176.194@KRBTEST.COM'} at 127.9.176.194:33205
I20260808 22:21:55.631140 9923 external_mini_cluster.cc:1428] Setting key aca5e12c808d91c9aac9db1d11104f1e
I20260808 22:21:55.637779 12549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:55.630648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.194:37361 (local address 127.9.176.254:44743)
0808 22:21:55.630770 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0808 22:21:55.630775 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:21:55.631356 (+ 581us) server_negotiation.cc:407] Connection header received
0808 22:21:55.631455 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:21:55.631458 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:21:55.631501 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:21:55.631571 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:21:55.632035 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:55.632750 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:55.633661 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:55.633868 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:55.633951 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:55.634898 (+ 947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:21:55.634901 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:21:55.634904 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:21:55.634926 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:21:55.636178 (+ 1252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:21:55.636614 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:21:55.636617 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:21:55.636619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:21:55.636695 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:21:55.636992 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:21:55.636995 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:21:55.636997 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:21:55.637203 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:21:55.637291 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0808 22:21:55.637536 (+ 245us) server_negotiation.cc:299] Negotiation successful
0808 22:21:55.637634 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260808 22:21:55.637794 13006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:55.630725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:44743 (local address 127.9.176.194:37361)
0808 22:21:55.631272 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0808 22:21:55.631286 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 22:21:55.631381 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:21:55.631606 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:21:55.631609 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:21:55.631792 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:21:55.631918 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:21:55.631925 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:21:55.632916 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:21:55.632920 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:21:55.633553 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:21:55.633562 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:21:55.633961 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:21:55.633964 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:21:55.634055 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:55.634469 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:21:55.634482 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:21:55.634776 (+ 294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:21:55.636298 (+ 1522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:21:55.636301 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:21:55.636303 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:21:55.636519 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:21:55.636778 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:21:55.636781 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:21:55.636783 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:21:55.636862 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:21:55.637299 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:21:55.637302 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:21:55.637388 (+ 86us) client_negotiation.cc:769] Sending connection context
0808 22:21:55.637483 (+ 95us) client_negotiation.cc:240] Negotiation successful
0808 22:21:55.637619 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"mutex_wait_us":3,"thread_start_us":86,"threads_started":1}
WARNING: no policy specified for kudu/127.9.176.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.176.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.176.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.195.keytab.
Entry for principal kudu/127.9.176.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.195.keytab.
I20260808 22:21:55.657492 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.195:0
--local_ip_for_outbound_sockets=127.9.176.195
--webserver_interface=127.9.176.195
--webserver_port=0
--tserver_master_addrs=127.9.176.254:44743
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.195.keytab
--principal=kudu/127.9.176.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:45665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:21:55.765529 13009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:21:55.765758 13009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:21:55.765799 13009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:21:55.769618 13009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:21:55.769666 13009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:21:55.769702 13009 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260808 22:21:55.769714 13009 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 22:21:55.769744 13009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.195
I20260808 22:21:55.774814 13009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:45665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.195.keytab
--principal=kudu/127.9.176.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.176.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.176.254:44743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.195
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:21:55.775743 13009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:21:55.776693 13009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:21:55.785964 13018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:21:55.785970 13014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:21:55.785970 13015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:21:55.786074 13009 server_base.cc:1034] running on GCE node
I20260808 22:21:55.786640 13009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:21:55.787226 13009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:21:55.788378 13009 hybrid_clock.cc:648] HybridClock initialized: now 1786227715788361 us; error 27 us; skew 500 ppm
Aug 08 22:21:55 dist-test-slave-ncp9 krb5kdc[12408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227715, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:21:55.790949 13009 init.cc:377] Logged in from keytab as kudu/127.9.176.195@KRBTEST.COM (short username kudu)
I20260808 22:21:55.791769 13009 webserver.cc:460] Webserver started at http://127.9.176.195:33451/ using document root <none> and password file <none>
I20260808 22:21:55.792258 13009 fs_manager.cc:359] Metadata directory not provided
I20260808 22:21:55.792313 13009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:21:55.792526 13009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:21:55.794477 13009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/minicluster-data/ts-2/data/instance:
uuid: "b9c4150963e04e018ba397151c05dc61"
format_stamp: "Formatted at 2026-08-08 22:21:55 on dist-test-slave-ncp9"
server_key: "4bd738b2e4660e005188909d39e7f1bc"
server_key_iv: "b438a3d9f0ff2977a8016c0850fbfba3"
server_key_version: "encryptionkey@0"
I20260808 22:21:55.794993 13009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/minicluster-data/ts-2/wal/instance:
uuid: "b9c4150963e04e018ba397151c05dc61"
format_stamp: "Formatted at 2026-08-08 22:21:55 on dist-test-slave-ncp9"
server_key: "4bd738b2e4660e005188909d39e7f1bc"
server_key_iv: "b438a3d9f0ff2977a8016c0850fbfba3"
server_key_version: "encryptionkey@0"
I20260808 22:21:55.798475 13009 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260808 22:21:55.800740 13034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:21:55.801896 13009 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260808 22:21:55.802001 13009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/minicluster-data/ts-2/wal
uuid: "b9c4150963e04e018ba397151c05dc61"
format_stamp: "Formatted at 2026-08-08 22:21:55 on dist-test-slave-ncp9"
server_key: "4bd738b2e4660e005188909d39e7f1bc"
server_key_iv: "b438a3d9f0ff2977a8016c0850fbfba3"
server_key_version: "encryptionkey@0"
I20260808 22:21:55.802089 13009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:21:55.812098 13009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:21:55.814700 13009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:21:55.814862 13009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:21:55.816174 13009 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:21:55.816223 13009 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:21:55.816273 13009 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:21:55.816301 13009 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 22:21:55 dist-test-slave-ncp9 krb5kdc[12408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227715, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.195@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
I20260808 22:21:55.826336 13009 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.195:45545
I20260808 22:21:55.826586 13208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.195:45545 every 8 connection(s)
I20260808 22:21:55.827319 13009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
I20260808 22:21:55.829013 12549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:55.816949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.195:50115 (local address 127.9.176.254:44743)
0808 22:21:55.817067 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0808 22:21:55.817071 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:21:55.818090 (+ 1019us) server_negotiation.cc:407] Connection header received
0808 22:21:55.819085 (+ 995us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:21:55.819089 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:21:55.819147 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:21:55.819204 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:21:55.820774 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:55.821389 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:55.822544 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:55.822846 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:55.822944 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:55.825481 (+ 2537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:21:55.825485 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:21:55.825487 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:21:55.825514 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:21:55.826956 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:21:55.827417 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:21:55.827420 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:21:55.827422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:21:55.827508 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:21:55.827898 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:21:55.827902 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:21:55.827903 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:21:55.828133 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:21:55.828230 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0808 22:21:55.828728 (+ 498us) server_negotiation.cc:299] Negotiation successful
0808 22:21:55.828845 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260808 22:21:55.829666 13052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:55.817436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:44743 (local address 127.9.176.195:50115)
0808 22:21:55.817935 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0808 22:21:55.817978 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 22:21:55.818830 (+ 852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:21:55.819643 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:21:55.819652 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:21:55.820015 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:21:55.820624 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:21:55.820646 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:21:55.821512 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:21:55.821515 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:21:55.822393 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:21:55.822402 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:21:55.823003 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:21:55.823008 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:21:55.823233 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:55.823831 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:21:55.823850 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:21:55.825315 (+ 1465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:21:55.827064 (+ 1749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:21:55.827069 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:21:55.827084 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:21:55.827322 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:21:55.827592 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:21:55.827595 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:21:55.827597 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:21:55.827786 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:21:55.828236 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:21:55.828243 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:21:55.828466 (+ 223us) client_negotiation.cc:769] Sending connection context
0808 22:21:55.828654 (+ 188us) client_negotiation.cc:240] Negotiation successful
0808 22:21:55.828928 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":135,"threads_started":1}
I20260808 22:21:55.830725 13210 heartbeater.cc:347] Connected to a master server at 127.9.176.254:44743
I20260808 22:21:55.830852 13210 heartbeater.cc:464] Registering TS with master...
I20260808 22:21:55.831310 13210 heartbeater.cc:511] Master 127.9.176.254:44743 requested a full tablet report, sending...
I20260808 22:21:55.832281 12464 ts_manager.cc:194] Registered new tserver with Master: b9c4150963e04e018ba397151c05dc61 (127.9.176.195:45545)
I20260808 22:21:55.832967 12464 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.176.195@KRBTEST.COM'} at 127.9.176.195:50115
I20260808 22:21:55.833125 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 13009
I20260808 22:21:55.833231 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/minicluster-data/ts-2/wal/instance
I20260808 22:21:55.833539 9923 external_mini_cluster.cc:1428] Setting key 61fd1298ce4c242a7ba2bab713cddb96
I20260808 22:21:55.835404 9923 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 22:21:55.839365 12549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:55.832752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.195:33125 (local address 127.9.176.254:44743)
0808 22:21:55.832846 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0808 22:21:55.832849 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:21:55.833027 (+ 178us) server_negotiation.cc:407] Connection header received
0808 22:21:55.833166 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:21:55.833168 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:21:55.833213 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:21:55.833276 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:21:55.833803 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:55.834364 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:55.835369 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:55.835609 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:55.835690 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:55.836754 (+ 1064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:21:55.836757 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:21:55.836759 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:21:55.836779 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:21:55.837952 (+ 1173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:21:55.838313 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:21:55.838317 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:21:55.838318 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:21:55.838396 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:21:55.838660 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:21:55.838663 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:21:55.838664 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:21:55.838821 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:21:55.838903 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0808 22:21:55.839145 (+ 242us) server_negotiation.cc:299] Negotiation successful
0808 22:21:55.839240 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260808 22:21:55.839375 13218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:55.832687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:44743 (local address 127.9.176.195:33125)
0808 22:21:55.832948 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0808 22:21:55.832960 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 22:21:55.833082 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:21:55.833384 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:21:55.833386 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:21:55.833562 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:21:55.833678 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:21:55.833685 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:21:55.834485 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:21:55.834488 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:21:55.835108 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:21:55.835117 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:21:55.835718 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:21:55.835722 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:21:55.835822 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:55.836245 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:21:55.836259 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:21:55.836633 (+ 374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:21:55.838044 (+ 1411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:21:55.838047 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:21:55.838048 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:21:55.838219 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:21:55.838500 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:21:55.838503 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:21:55.838505 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:21:55.838579 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:21:55.838908 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:21:55.838911 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:21:55.839000 (+ 89us) client_negotiation.cc:769] Sending connection context
0808 22:21:55.839083 (+ 83us) client_negotiation.cc:240] Negotiation successful
0808 22:21:55.839209 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":78,"threads_started":1}
Aug 08 22:21:55 dist-test-slave-ncp9 krb5kdc[12408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227715, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 22:21:55 dist-test-slave-ncp9 krb5kdc[12408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227715, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
I20260808 22:21:55.862109 12549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:55.854480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40622 (local address 127.9.176.254:44743)
0808 22:21:55.854617 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0808 22:21:55.854621 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:21:55.854634 (+ 13us) server_negotiation.cc:407] Connection header received
0808 22:21:55.854671 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:21:55.854673 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:21:55.854708 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:21:55.854774 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:21:55.855469 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:55.855971 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:55.856801 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:55.857047 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:55.857144 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:55.859339 (+ 2195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:21:55.859343 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:21:55.859346 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:21:55.859369 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:21:55.860603 (+ 1234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:21:55.861013 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:21:55.861017 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:21:55.861018 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:21:55.861092 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:21:55.861343 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:21:55.861347 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:21:55.861348 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:21:55.861514 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:21:55.861628 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0808 22:21:55.861861 (+ 233us) server_negotiation.cc:299] Negotiation successful
0808 22:21:55.861964 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 22:21:55.864635 12464 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40622:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 22:21:55.867563 12464 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 22:21:55.881562 13241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:55.875570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.193:34469 (local address 127.0.0.1:45434)
0808 22:21:55.876421 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0808 22:21:55.876434 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 22:21:55.876544 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:21:55.877235 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:21:55.877238 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:21:55.877258 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:21:55.877385 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:21:55.877392 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:21:55.878865 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:21:55.878868 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:21:55.880170 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:21:55.880181 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:21:55.881076 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:21:55.881080 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:21:55.881194 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:55.881233 (+ 39us) client_negotiation.cc:769] Sending connection context
0808 22:21:55.881334 (+ 101us) client_negotiation.cc:240] Negotiation successful
0808 22:21:55.881430 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":792,"mutex_wait_us":36,"thread_start_us":97,"threads_started":1}
I20260808 22:21:55.881767 13243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:55.876247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50896 (local address 127.9.176.195:45545)
0808 22:21:55.876724 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0808 22:21:55.876734 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 22:21:55.877148 (+ 414us) server_negotiation.cc:407] Connection header received
0808 22:21:55.877557 (+ 409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:21:55.877562 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:21:55.877730 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:21:55.877872 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:21:55.878095 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:55.879259 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:55.880487 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:55.881092 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:55.881191 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:55.881322 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0808 22:21:55.881528 (+ 206us) server_negotiation.cc:299] Negotiation successful
0808 22:21:55.881622 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":111,"threads_started":1}
I20260808 22:21:55.881896 13242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:55.876207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45434 (local address 127.9.176.193:34469)
0808 22:21:55.876783 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0808 22:21:55.876795 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 22:21:55.876817 (+ 22us) server_negotiation.cc:407] Connection header received
0808 22:21:55.876920 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:21:55.876926 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:21:55.877105 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:21:55.877263 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:21:55.877502 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:55.878730 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:55.880301 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:55.880938 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:55.881043 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:55.881203 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0808 22:21:55.881642 (+ 439us) server_negotiation.cc:299] Negotiation successful
0808 22:21:55.881735 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":103,"threads_started":1}
I20260808 22:21:55.882311 13236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:55.875438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.194:36839 (local address 127.0.0.1:46780)
0808 22:21:55.875908 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0808 22:21:55.875952 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 22:21:55.876071 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:21:55.877219 (+ 1148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:21:55.877225 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:21:55.877246 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:21:55.877350 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:21:55.877354 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:21:55.878534 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:21:55.878538 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:21:55.880170 (+ 1632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:21:55.880181 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:21:55.881910 (+ 1729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:21:55.881914 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:21:55.882000 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:55.882024 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 22:21:55.882078 (+ 54us) client_negotiation.cc:240] Negotiation successful
0808 22:21:55.882138 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"mutex_wait_us":76,"thread_start_us":77,"threads_started":1}
I20260808 22:21:55.882167 13239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:55.875993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.195:45545 (local address 127.0.0.1:50896)
0808 22:21:55.877033 (+ 1040us) negotiation.cc:107] Waiting for socket to connect
0808 22:21:55.877046 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 22:21:55.877157 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:21:55.877874 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:21:55.877877 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:21:55.877896 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:21:55.877991 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:21:55.877996 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:21:55.879377 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:21:55.879380 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:21:55.880348 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:21:55.880359 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:21:55.881283 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:21:55.881286 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:21:55.881385 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:55.881409 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 22:21:55.881485 (+ 76us) client_negotiation.cc:240] Negotiation successful
0808 22:21:55.881549 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":980,"mutex_wait_us":23,"thread_start_us":91,"threads_started":1}
I20260808 22:21:55.882969 13240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:55.875567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46780 (local address 127.9.176.194:36839)
0808 22:21:55.876692 (+ 1125us) server_negotiation.cc:206] Beginning negotiation
0808 22:21:55.876704 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 22:21:55.876729 (+ 25us) server_negotiation.cc:407] Connection header received
0808 22:21:55.876809 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:21:55.876814 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:21:55.877055 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:21:55.877468 (+ 413us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:21:55.877572 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:55.878401 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:55.880994 (+ 2593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:55.881771 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:55.882564 (+ 793us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:55.882692 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0808 22:21:55.882738 (+ 46us) server_negotiation.cc:299] Negotiation successful
0808 22:21:55.882818 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1008,"thread_start_us":107,"threads_started":1}
I20260808 22:21:55.884917 12681 tablet_service.cc:1467] Processing CreateTablet for tablet 319c06acd55a4d6bbde8caf328e8c21a (DEFAULT_TABLE table=test-table [id=9fb1111cbf154052bf145668cd6fdb93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:21:55.885154 12904 tablet_service.cc:1467] Processing CreateTablet for tablet 319c06acd55a4d6bbde8caf328e8c21a (DEFAULT_TABLE table=test-table [id=9fb1111cbf154052bf145668cd6fdb93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:21:55.886016 12904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 319c06acd55a4d6bbde8caf328e8c21a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:21:55.886037 12681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 319c06acd55a4d6bbde8caf328e8c21a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:21:55.886706 13113 tablet_service.cc:1467] Processing CreateTablet for tablet 319c06acd55a4d6bbde8caf328e8c21a (DEFAULT_TABLE table=test-table [id=9fb1111cbf154052bf145668cd6fdb93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:21:55.887617 13113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 319c06acd55a4d6bbde8caf328e8c21a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:21:55.892443 13244 tablet_bootstrap.cc:492] T 319c06acd55a4d6bbde8caf328e8c21a P bcba8d834ed44491a94b32067ba681ad: Bootstrap starting.
I20260808 22:21:55.893921 13245 tablet_bootstrap.cc:492] T 319c06acd55a4d6bbde8caf328e8c21a P ec50da59889f4a1e84a835ff268b75e8: Bootstrap starting.
I20260808 22:21:55.894647 13246 tablet_bootstrap.cc:492] T 319c06acd55a4d6bbde8caf328e8c21a P b9c4150963e04e018ba397151c05dc61: Bootstrap starting.
I20260808 22:21:55.894836 13244 tablet_bootstrap.cc:654] T 319c06acd55a4d6bbde8caf328e8c21a P bcba8d834ed44491a94b32067ba681ad: Neither blocks nor log segments found. Creating new log.
I20260808 22:21:55.895690 13244 log.cc:824] T 319c06acd55a4d6bbde8caf328e8c21a P bcba8d834ed44491a94b32067ba681ad: Log is configured to *not* fsync() on all Append() calls
I20260808 22:21:55.896008 13245 tablet_bootstrap.cc:654] T 319c06acd55a4d6bbde8caf328e8c21a P ec50da59889f4a1e84a835ff268b75e8: Neither blocks nor log segments found. Creating new log.
I20260808 22:21:55.896727 13246 tablet_bootstrap.cc:654] T 319c06acd55a4d6bbde8caf328e8c21a P b9c4150963e04e018ba397151c05dc61: Neither blocks nor log segments found. Creating new log.
I20260808 22:21:55.896764 13245 log.cc:824] T 319c06acd55a4d6bbde8caf328e8c21a P ec50da59889f4a1e84a835ff268b75e8: Log is configured to *not* fsync() on all Append() calls
I20260808 22:21:55.897500 13246 log.cc:824] T 319c06acd55a4d6bbde8caf328e8c21a P b9c4150963e04e018ba397151c05dc61: Log is configured to *not* fsync() on all Append() calls
I20260808 22:21:55.897728 13244 tablet_bootstrap.cc:492] T 319c06acd55a4d6bbde8caf328e8c21a P bcba8d834ed44491a94b32067ba681ad: No bootstrap required, opened a new log
I20260808 22:21:55.897919 13244 ts_tablet_manager.cc:1397] T 319c06acd55a4d6bbde8caf328e8c21a P bcba8d834ed44491a94b32067ba681ad: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260808 22:21:55.898528 13245 tablet_bootstrap.cc:492] T 319c06acd55a4d6bbde8caf328e8c21a P ec50da59889f4a1e84a835ff268b75e8: No bootstrap required, opened a new log
I20260808 22:21:55.898710 13245 ts_tablet_manager.cc:1397] T 319c06acd55a4d6bbde8caf328e8c21a P ec50da59889f4a1e84a835ff268b75e8: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260808 22:21:55.899346 13246 tablet_bootstrap.cc:492] T 319c06acd55a4d6bbde8caf328e8c21a P b9c4150963e04e018ba397151c05dc61: No bootstrap required, opened a new log
I20260808 22:21:55.899526 13246 ts_tablet_manager.cc:1397] T 319c06acd55a4d6bbde8caf328e8c21a P b9c4150963e04e018ba397151c05dc61: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260808 22:21:55.901646 13245 raft_consensus.cc:348] T 319c06acd55a4d6bbde8caf328e8c21a P ec50da59889f4a1e84a835ff268b75e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcba8d834ed44491a94b32067ba681ad" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36839 } } peers { permanent_uuid: "ec50da59889f4a1e84a835ff268b75e8" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34469 } } peers { permanent_uuid: "b9c4150963e04e018ba397151c05dc61" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45545 } }
I20260808 22:21:55.901871 13245 raft_consensus.cc:374] T 319c06acd55a4d6bbde8caf328e8c21a P ec50da59889f4a1e84a835ff268b75e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:21:55.901666 13244 raft_consensus.cc:348] T 319c06acd55a4d6bbde8caf328e8c21a P bcba8d834ed44491a94b32067ba681ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcba8d834ed44491a94b32067ba681ad" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36839 } } peers { permanent_uuid: "ec50da59889f4a1e84a835ff268b75e8" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34469 } } peers { permanent_uuid: "b9c4150963e04e018ba397151c05dc61" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45545 } }
I20260808 22:21:55.901922 13245 raft_consensus.cc:729] T 319c06acd55a4d6bbde8caf328e8c21a P ec50da59889f4a1e84a835ff268b75e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec50da59889f4a1e84a835ff268b75e8, State: Initialized, Role: FOLLOWER
I20260808 22:21:55.901970 13244 raft_consensus.cc:374] T 319c06acd55a4d6bbde8caf328e8c21a P bcba8d834ed44491a94b32067ba681ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:21:55.902027 13244 raft_consensus.cc:729] T 319c06acd55a4d6bbde8caf328e8c21a P bcba8d834ed44491a94b32067ba681ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcba8d834ed44491a94b32067ba681ad, State: Initialized, Role: FOLLOWER
I20260808 22:21:55.902349 13245 consensus_queue.cc:260] T 319c06acd55a4d6bbde8caf328e8c21a P ec50da59889f4a1e84a835ff268b75e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcba8d834ed44491a94b32067ba681ad" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36839 } } peers { permanent_uuid: "ec50da59889f4a1e84a835ff268b75e8" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34469 } } peers { permanent_uuid: "b9c4150963e04e018ba397151c05dc61" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45545 } }
I20260808 22:21:55.902503 13244 consensus_queue.cc:260] T 319c06acd55a4d6bbde8caf328e8c21a P bcba8d834ed44491a94b32067ba681ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcba8d834ed44491a94b32067ba681ad" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36839 } } peers { permanent_uuid: "ec50da59889f4a1e84a835ff268b75e8" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34469 } } peers { permanent_uuid: "b9c4150963e04e018ba397151c05dc61" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45545 } }
I20260808 22:21:55.903023 12786 heartbeater.cc:503] Master 127.9.176.254:44743 was elected leader, sending a full tablet report...
I20260808 22:21:55.903255 13246 raft_consensus.cc:348] T 319c06acd55a4d6bbde8caf328e8c21a P b9c4150963e04e018ba397151c05dc61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcba8d834ed44491a94b32067ba681ad" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36839 } } peers { permanent_uuid: "ec50da59889f4a1e84a835ff268b75e8" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34469 } } peers { permanent_uuid: "b9c4150963e04e018ba397151c05dc61" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45545 } }
I20260808 22:21:55.903484 13244 ts_tablet_manager.cc:1428] T 319c06acd55a4d6bbde8caf328e8c21a P bcba8d834ed44491a94b32067ba681ad: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260808 22:21:55.903488 13246 raft_consensus.cc:374] T 319c06acd55a4d6bbde8caf328e8c21a P b9c4150963e04e018ba397151c05dc61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:21:55.903538 13246 raft_consensus.cc:729] T 319c06acd55a4d6bbde8caf328e8c21a P b9c4150963e04e018ba397151c05dc61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9c4150963e04e018ba397151c05dc61, State: Initialized, Role: FOLLOWER
I20260808 22:21:55.903808 12997 heartbeater.cc:503] Master 127.9.176.254:44743 was elected leader, sending a full tablet report...
I20260808 22:21:55.903934 13246 consensus_queue.cc:260] T 319c06acd55a4d6bbde8caf328e8c21a P b9c4150963e04e018ba397151c05dc61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcba8d834ed44491a94b32067ba681ad" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36839 } } peers { permanent_uuid: "ec50da59889f4a1e84a835ff268b75e8" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34469 } } peers { permanent_uuid: "b9c4150963e04e018ba397151c05dc61" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45545 } }
I20260808 22:21:55.904145 13245 ts_tablet_manager.cc:1428] T 319c06acd55a4d6bbde8caf328e8c21a P ec50da59889f4a1e84a835ff268b75e8: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260808 22:21:55.904634 13210 heartbeater.cc:503] Master 127.9.176.254:44743 was elected leader, sending a full tablet report...
I20260808 22:21:55.905392 13246 ts_tablet_manager.cc:1428] T 319c06acd55a4d6bbde8caf328e8c21a P b9c4150963e04e018ba397151c05dc61: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260808 22:21:55.920186 12788 tablet.cc:2369] T 319c06acd55a4d6bbde8caf328e8c21a P ec50da59889f4a1e84a835ff268b75e8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 22:21:55.972949 13255 raft_consensus.cc:482] T 319c06acd55a4d6bbde8caf328e8c21a P b9c4150963e04e018ba397151c05dc61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 22:21:55.973171 13255 raft_consensus.cc:504] T 319c06acd55a4d6bbde8caf328e8c21a P b9c4150963e04e018ba397151c05dc61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcba8d834ed44491a94b32067ba681ad" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36839 } } peers { permanent_uuid: "ec50da59889f4a1e84a835ff268b75e8" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34469 } } peers { permanent_uuid: "b9c4150963e04e018ba397151c05dc61" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45545 } }
I20260808 22:21:55.974290 13255 leader_election.cc:288] T 319c06acd55a4d6bbde8caf328e8c21a P b9c4150963e04e018ba397151c05dc61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bcba8d834ed44491a94b32067ba681ad (127.9.176.194:36839), ec50da59889f4a1e84a835ff268b75e8 (127.9.176.193:34469)
I20260808 22:21:55.978502 13242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:55.974613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.195:55675 (local address 127.9.176.193:34469)
0808 22:21:55.974726 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0808 22:21:55.974730 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:21:55.974853 (+ 123us) server_negotiation.cc:407] Connection header received
0808 22:21:55.974915 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:21:55.974917 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:21:55.974957 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:21:55.975021 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:21:55.975273 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:55.976035 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:55.977412 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:55.978017 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:55.978136 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:55.978194 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0808 22:21:55.978343 (+ 149us) server_negotiation.cc:299] Negotiation successful
0808 22:21:55.978401 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 22:21:55.978539 13052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:55.974548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.193:34469 (local address 127.9.176.195:55675)
0808 22:21:55.974727 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0808 22:21:55.974742 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 22:21:55.974838 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:21:55.975048 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:21:55.975051 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:21:55.975066 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:21:55.975182 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:21:55.975188 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:21:55.976137 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:21:55.976140 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:21:55.977297 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:21:55.977306 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:21:55.978124 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:21:55.978127 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:21:55.978230 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:55.978256 (+ 26us) client_negotiation.cc:769] Sending connection context
0808 22:21:55.978332 (+ 76us) client_negotiation.cc:240] Negotiation successful
0808 22:21:55.978400 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95}
I20260808 22:21:55.979228 12716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "319c06acd55a4d6bbde8caf328e8c21a" candidate_uuid: "b9c4150963e04e018ba397151c05dc61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec50da59889f4a1e84a835ff268b75e8" is_pre_election: true
I20260808 22:21:55.979358 13256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:55.974549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.194:36839 (local address 127.9.176.195:52843)
0808 22:21:55.974916 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0808 22:21:55.974927 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 22:21:55.975013 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:21:55.975388 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:21:55.975391 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:21:55.975409 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:21:55.975976 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:21:55.975986 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:21:55.977048 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:21:55.977051 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:21:55.978086 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:21:55.978096 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:21:55.979046 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:21:55.979049 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:21:55.979130 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:55.979148 (+ 18us) client_negotiation.cc:769] Sending connection context
0808 22:21:55.979207 (+ 59us) client_negotiation.cc:240] Negotiation successful
0808 22:21:55.979254 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"mutex_wait_us":30,"thread_start_us":120,"threads_started":1}
I20260808 22:21:55.979406 13240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:55.974773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.195:52843 (local address 127.9.176.194:36839)
0808 22:21:55.975174 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0808 22:21:55.975178 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:21:55.975191 (+ 13us) server_negotiation.cc:407] Connection header received
0808 22:21:55.975238 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:21:55.975242 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:21:55.975286 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:21:55.975343 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:21:55.976121 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:55.976940 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:55.978216 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:55.978922 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:55.979041 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:55.979086 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0808 22:21:55.979234 (+ 148us) server_negotiation.cc:299] Negotiation successful
0808 22:21:55.979300 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319}
I20260808 22:21:55.979597 12716 raft_consensus.cc:2454] T 319c06acd55a4d6bbde8caf328e8c21a P ec50da59889f4a1e84a835ff268b75e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9c4150963e04e018ba397151c05dc61 in term 0.
I20260808 22:21:55.980065 13045 leader_election.cc:302] T 319c06acd55a4d6bbde8caf328e8c21a P b9c4150963e04e018ba397151c05dc61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9c4150963e04e018ba397151c05dc61, ec50da59889f4a1e84a835ff268b75e8; no voters:
I20260808 22:21:55.980080 12928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "319c06acd55a4d6bbde8caf328e8c21a" candidate_uuid: "b9c4150963e04e018ba397151c05dc61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcba8d834ed44491a94b32067ba681ad" is_pre_election: true
I20260808 22:21:55.980324 12928 raft_consensus.cc:2454] T 319c06acd55a4d6bbde8caf328e8c21a P bcba8d834ed44491a94b32067ba681ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9c4150963e04e018ba397151c05dc61 in term 0.
I20260808 22:21:55.980341 13255 raft_consensus.cc:2781] T 319c06acd55a4d6bbde8caf328e8c21a P b9c4150963e04e018ba397151c05dc61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 22:21:55.980418 13255 raft_consensus.cc:482] T 319c06acd55a4d6bbde8caf328e8c21a P b9c4150963e04e018ba397151c05dc61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 22:21:55.980460 13255 raft_consensus.cc:3037] T 319c06acd55a4d6bbde8caf328e8c21a P b9c4150963e04e018ba397151c05dc61 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:21:55.981549 13255 raft_consensus.cc:504] T 319c06acd55a4d6bbde8caf328e8c21a P b9c4150963e04e018ba397151c05dc61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcba8d834ed44491a94b32067ba681ad" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36839 } } peers { permanent_uuid: "ec50da59889f4a1e84a835ff268b75e8" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34469 } } peers { permanent_uuid: "b9c4150963e04e018ba397151c05dc61" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45545 } }
I20260808 22:21:55.981858 13255 leader_election.cc:288] T 319c06acd55a4d6bbde8caf328e8c21a P b9c4150963e04e018ba397151c05dc61 [CANDIDATE]: Term 1 election: Requested vote from peers bcba8d834ed44491a94b32067ba681ad (127.9.176.194:36839), ec50da59889f4a1e84a835ff268b75e8 (127.9.176.193:34469)
I20260808 22:21:55.982177 12928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "319c06acd55a4d6bbde8caf328e8c21a" candidate_uuid: "b9c4150963e04e018ba397151c05dc61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcba8d834ed44491a94b32067ba681ad"
I20260808 22:21:55.982177 12716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "319c06acd55a4d6bbde8caf328e8c21a" candidate_uuid: "b9c4150963e04e018ba397151c05dc61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec50da59889f4a1e84a835ff268b75e8"
I20260808 22:21:55.982306 12716 raft_consensus.cc:3037] T 319c06acd55a4d6bbde8caf328e8c21a P ec50da59889f4a1e84a835ff268b75e8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:21:55.982306 12928 raft_consensus.cc:3037] T 319c06acd55a4d6bbde8caf328e8c21a P bcba8d834ed44491a94b32067ba681ad [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:21:55.983465 12716 raft_consensus.cc:2454] T 319c06acd55a4d6bbde8caf328e8c21a P ec50da59889f4a1e84a835ff268b75e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9c4150963e04e018ba397151c05dc61 in term 1.
I20260808 22:21:55.983465 12928 raft_consensus.cc:2454] T 319c06acd55a4d6bbde8caf328e8c21a P bcba8d834ed44491a94b32067ba681ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9c4150963e04e018ba397151c05dc61 in term 1.
I20260808 22:21:55.983754 13045 leader_election.cc:302] T 319c06acd55a4d6bbde8caf328e8c21a P b9c4150963e04e018ba397151c05dc61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: b9c4150963e04e018ba397151c05dc61, bcba8d834ed44491a94b32067ba681ad, ec50da59889f4a1e84a835ff268b75e8; no voters:
I20260808 22:21:55.983918 13255 raft_consensus.cc:2781] T 319c06acd55a4d6bbde8caf328e8c21a P b9c4150963e04e018ba397151c05dc61 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:21:55.984124 13255 raft_consensus.cc:686] T 319c06acd55a4d6bbde8caf328e8c21a P b9c4150963e04e018ba397151c05dc61 [term 1 LEADER]: Becoming Leader. State: Replica: b9c4150963e04e018ba397151c05dc61, State: Running, Role: LEADER
I20260808 22:21:55.984440 13255 consensus_queue.cc:237] T 319c06acd55a4d6bbde8caf328e8c21a P b9c4150963e04e018ba397151c05dc61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcba8d834ed44491a94b32067ba681ad" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36839 } } peers { permanent_uuid: "ec50da59889f4a1e84a835ff268b75e8" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34469 } } peers { permanent_uuid: "b9c4150963e04e018ba397151c05dc61" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45545 } }
I20260808 22:21:55.987468 12463 catalog_manager.cc:5495] T 319c06acd55a4d6bbde8caf328e8c21a P b9c4150963e04e018ba397151c05dc61 reported cstate change: term changed from 0 to 1, leader changed from <none> to b9c4150963e04e018ba397151c05dc61 (127.9.176.195). New cstate: current_term: 1 leader_uuid: "b9c4150963e04e018ba397151c05dc61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcba8d834ed44491a94b32067ba681ad" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36839 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec50da59889f4a1e84a835ff268b75e8" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9c4150963e04e018ba397151c05dc61" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45545 } health_report { overall_health: HEALTHY } } }
Aug 08 22:21:56 dist-test-slave-ncp9 krb5kdc[12408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227715, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.254@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
I20260808 22:21:56.026410 12549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:56.018652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40630 (local address 127.9.176.254:44743)
0808 22:21:56.018779 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0808 22:21:56.018784 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:21:56.018928 (+ 144us) server_negotiation.cc:407] Connection header received
0808 22:21:56.019060 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:21:56.019063 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:21:56.019102 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:21:56.019167 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:21:56.019874 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:56.020384 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:56.021196 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:56.021399 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:56.021483 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:56.023617 (+ 2134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:21:56.023619 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:21:56.023621 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:21:56.023641 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:21:56.024815 (+ 1174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:21:56.025262 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:21:56.025265 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:21:56.025266 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:21:56.025337 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:21:56.025571 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:21:56.025575 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:21:56.025577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:21:56.025734 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:21:56.025817 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0808 22:21:56.026097 (+ 280us) server_negotiation.cc:299] Negotiation successful
0808 22:21:56.026246 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":74,"server-negotiator.queue_time_us":43}
I20260808 22:21:56.026420 13267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:56.018548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:44743 (local address 127.0.0.1:40630)
0808 22:21:56.018846 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0808 22:21:56.018859 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 22:21:56.018974 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:21:56.019199 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:21:56.019202 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:21:56.019659 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:21:56.019761 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:21:56.019768 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:21:56.020482 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:21:56.020485 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:21:56.021092 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:21:56.021102 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:21:56.021491 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:21:56.021494 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:21:56.021584 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:56.022083 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:21:56.022095 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:21:56.023479 (+ 1384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:21:56.024931 (+ 1452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:21:56.024940 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:21:56.024946 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:21:56.025175 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:21:56.025415 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:21:56.025418 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:21:56.025419 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:21:56.025490 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:21:56.025833 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:21:56.025841 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:21:56.025943 (+ 102us) client_negotiation.cc:769] Sending connection context
0808 22:21:56.026037 (+ 94us) client_negotiation.cc:240] Negotiation successful
0808 22:21:56.026196 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"mutex_wait_us":1,"thread_start_us":117,"threads_started":1}
I20260808 22:21:56.030256 12463 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.9.176.254@KRBTEST.COM'} at 127.0.0.1:40630:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260808 22:21:56.030925 12463 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 22:21:56.035531 12681 tablet_service.cc:1467] Processing CreateTablet for tablet e7b59f47cc0648149772dee7db2eb681 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d65a9794d57f425b9774ec6b84c9ae89]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 22:21:56.035729 12904 tablet_service.cc:1467] Processing CreateTablet for tablet e7b59f47cc0648149772dee7db2eb681 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d65a9794d57f425b9774ec6b84c9ae89]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 22:21:56.035706 13113 tablet_service.cc:1467] Processing CreateTablet for tablet e7b59f47cc0648149772dee7db2eb681 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d65a9794d57f425b9774ec6b84c9ae89]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 22:21:56.036300 12904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7b59f47cc0648149772dee7db2eb681. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:21:56.036300 12681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7b59f47cc0648149772dee7db2eb681. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:21:56.036396 13113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7b59f47cc0648149772dee7db2eb681. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:21:56.039026 13246 tablet_bootstrap.cc:492] T e7b59f47cc0648149772dee7db2eb681 P b9c4150963e04e018ba397151c05dc61: Bootstrap starting.
I20260808 22:21:56.039141 13245 tablet_bootstrap.cc:492] T e7b59f47cc0648149772dee7db2eb681 P ec50da59889f4a1e84a835ff268b75e8: Bootstrap starting.
I20260808 22:21:56.039227 13244 tablet_bootstrap.cc:492] T e7b59f47cc0648149772dee7db2eb681 P bcba8d834ed44491a94b32067ba681ad: Bootstrap starting.
I20260808 22:21:56.039896 13245 tablet_bootstrap.cc:654] T e7b59f47cc0648149772dee7db2eb681 P ec50da59889f4a1e84a835ff268b75e8: Neither blocks nor log segments found. Creating new log.
I20260808 22:21:56.040011 13246 tablet_bootstrap.cc:654] T e7b59f47cc0648149772dee7db2eb681 P b9c4150963e04e018ba397151c05dc61: Neither blocks nor log segments found. Creating new log.
I20260808 22:21:56.040072 13244 tablet_bootstrap.cc:654] T e7b59f47cc0648149772dee7db2eb681 P bcba8d834ed44491a94b32067ba681ad: Neither blocks nor log segments found. Creating new log.
I20260808 22:21:56.041052 13245 tablet_bootstrap.cc:492] T e7b59f47cc0648149772dee7db2eb681 P ec50da59889f4a1e84a835ff268b75e8: No bootstrap required, opened a new log
I20260808 22:21:56.041146 13245 ts_tablet_manager.cc:1397] T e7b59f47cc0648149772dee7db2eb681 P ec50da59889f4a1e84a835ff268b75e8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 22:21:56.041591 13246 tablet_bootstrap.cc:492] T e7b59f47cc0648149772dee7db2eb681 P b9c4150963e04e018ba397151c05dc61: No bootstrap required, opened a new log
I20260808 22:21:56.041679 13246 ts_tablet_manager.cc:1397] T e7b59f47cc0648149772dee7db2eb681 P b9c4150963e04e018ba397151c05dc61: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 22:21:56.041679 13244 tablet_bootstrap.cc:492] T e7b59f47cc0648149772dee7db2eb681 P bcba8d834ed44491a94b32067ba681ad: No bootstrap required, opened a new log
I20260808 22:21:56.041623 13245 raft_consensus.cc:348] T e7b59f47cc0648149772dee7db2eb681 P ec50da59889f4a1e84a835ff268b75e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec50da59889f4a1e84a835ff268b75e8" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34469 } } peers { permanent_uuid: "bcba8d834ed44491a94b32067ba681ad" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36839 } } peers { permanent_uuid: "b9c4150963e04e018ba397151c05dc61" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45545 } }
I20260808 22:21:56.041726 13245 raft_consensus.cc:374] T e7b59f47cc0648149772dee7db2eb681 P ec50da59889f4a1e84a835ff268b75e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:21:56.041757 13244 ts_tablet_manager.cc:1397] T e7b59f47cc0648149772dee7db2eb681 P bcba8d834ed44491a94b32067ba681ad: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260808 22:21:56.041770 13245 raft_consensus.cc:729] T e7b59f47cc0648149772dee7db2eb681 P ec50da59889f4a1e84a835ff268b75e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec50da59889f4a1e84a835ff268b75e8, State: Initialized, Role: FOLLOWER
I20260808 22:21:56.041898 13245 consensus_queue.cc:260] T e7b59f47cc0648149772dee7db2eb681 P ec50da59889f4a1e84a835ff268b75e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec50da59889f4a1e84a835ff268b75e8" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34469 } } peers { permanent_uuid: "bcba8d834ed44491a94b32067ba681ad" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36839 } } peers { permanent_uuid: "b9c4150963e04e018ba397151c05dc61" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45545 } }
I20260808 22:21:56.042093 13246 raft_consensus.cc:348] T e7b59f47cc0648149772dee7db2eb681 P b9c4150963e04e018ba397151c05dc61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec50da59889f4a1e84a835ff268b75e8" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34469 } } peers { permanent_uuid: "bcba8d834ed44491a94b32067ba681ad" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36839 } } peers { permanent_uuid: "b9c4150963e04e018ba397151c05dc61" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45545 } }
I20260808 22:21:56.042150 13245 ts_tablet_manager.cc:1428] T e7b59f47cc0648149772dee7db2eb681 P ec50da59889f4a1e84a835ff268b75e8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 22:21:56.042197 13246 raft_consensus.cc:374] T e7b59f47cc0648149772dee7db2eb681 P b9c4150963e04e018ba397151c05dc61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:21:56.042237 13246 raft_consensus.cc:729] T e7b59f47cc0648149772dee7db2eb681 P b9c4150963e04e018ba397151c05dc61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9c4150963e04e018ba397151c05dc61, State: Initialized, Role: FOLLOWER
I20260808 22:21:56.042201 13244 raft_consensus.cc:348] T e7b59f47cc0648149772dee7db2eb681 P bcba8d834ed44491a94b32067ba681ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec50da59889f4a1e84a835ff268b75e8" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34469 } } peers { permanent_uuid: "bcba8d834ed44491a94b32067ba681ad" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36839 } } peers { permanent_uuid: "b9c4150963e04e018ba397151c05dc61" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45545 } }
I20260808 22:21:56.042309 13244 raft_consensus.cc:374] T e7b59f47cc0648149772dee7db2eb681 P bcba8d834ed44491a94b32067ba681ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:21:56.042346 13244 raft_consensus.cc:729] T e7b59f47cc0648149772dee7db2eb681 P bcba8d834ed44491a94b32067ba681ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcba8d834ed44491a94b32067ba681ad, State: Initialized, Role: FOLLOWER
I20260808 22:21:56.042356 13246 consensus_queue.cc:260] T e7b59f47cc0648149772dee7db2eb681 P b9c4150963e04e018ba397151c05dc61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec50da59889f4a1e84a835ff268b75e8" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34469 } } peers { permanent_uuid: "bcba8d834ed44491a94b32067ba681ad" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36839 } } peers { permanent_uuid: "b9c4150963e04e018ba397151c05dc61" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45545 } }
I20260808 22:21:56.042384 13251 tablet_replica.cc:439] T e7b59f47cc0648149772dee7db2eb681 P ec50da59889f4a1e84a835ff268b75e8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec50da59889f4a1e84a835ff268b75e8" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34469 } } peers { permanent_uuid: "bcba8d834ed44491a94b32067ba681ad" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36839 } } peers { permanent_uuid: "b9c4150963e04e018ba397151c05dc61" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45545 } } }
I20260808 22:21:56.042466 13244 consensus_queue.cc:260] T e7b59f47cc0648149772dee7db2eb681 P bcba8d834ed44491a94b32067ba681ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec50da59889f4a1e84a835ff268b75e8" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34469 } } peers { permanent_uuid: "bcba8d834ed44491a94b32067ba681ad" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36839 } } peers { permanent_uuid: "b9c4150963e04e018ba397151c05dc61" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45545 } }
I20260808 22:21:56.042520 13251 tablet_replica.cc:442] T e7b59f47cc0648149772dee7db2eb681 P ec50da59889f4a1e84a835ff268b75e8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 22:21:56.042627 13246 ts_tablet_manager.cc:1428] T e7b59f47cc0648149772dee7db2eb681 P b9c4150963e04e018ba397151c05dc61: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 22:21:56.042728 13244 ts_tablet_manager.cc:1428] T e7b59f47cc0648149772dee7db2eb681 P bcba8d834ed44491a94b32067ba681ad: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260808 22:21:56.042719 13253 tablet_replica.cc:439] T e7b59f47cc0648149772dee7db2eb681 P bcba8d834ed44491a94b32067ba681ad: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec50da59889f4a1e84a835ff268b75e8" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34469 } } peers { permanent_uuid: "bcba8d834ed44491a94b32067ba681ad" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36839 } } peers { permanent_uuid: "b9c4150963e04e018ba397151c05dc61" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45545 } } }
I20260808 22:21:56.042809 13255 tablet_replica.cc:439] T e7b59f47cc0648149772dee7db2eb681 P b9c4150963e04e018ba397151c05dc61: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec50da59889f4a1e84a835ff268b75e8" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34469 } } peers { permanent_uuid: "bcba8d834ed44491a94b32067ba681ad" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36839 } } peers { permanent_uuid: "b9c4150963e04e018ba397151c05dc61" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45545 } } }
I20260808 22:21:56.042863 13253 tablet_replica.cc:442] T e7b59f47cc0648149772dee7db2eb681 P bcba8d834ed44491a94b32067ba681ad: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 22:21:56.042904 13255 tablet_replica.cc:442] T e7b59f47cc0648149772dee7db2eb681 P b9c4150963e04e018ba397151c05dc61: This TxnStatusTablet replica's current role is: FOLLOWER
W20260808 22:21:56.078218 13211 tablet.cc:2369] T 319c06acd55a4d6bbde8caf328e8c21a P b9c4150963e04e018ba397151c05dc61: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 22:21:56.082227 13251 raft_consensus.cc:482] T e7b59f47cc0648149772dee7db2eb681 P ec50da59889f4a1e84a835ff268b75e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 22:21:56.082346 13251 raft_consensus.cc:504] T e7b59f47cc0648149772dee7db2eb681 P ec50da59889f4a1e84a835ff268b75e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec50da59889f4a1e84a835ff268b75e8" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34469 } } peers { permanent_uuid: "bcba8d834ed44491a94b32067ba681ad" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36839 } } peers { permanent_uuid: "b9c4150963e04e018ba397151c05dc61" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45545 } }
I20260808 22:21:56.083395 13251 leader_election.cc:288] T e7b59f47cc0648149772dee7db2eb681 P ec50da59889f4a1e84a835ff268b75e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bcba8d834ed44491a94b32067ba681ad (127.9.176.194:36839), b9c4150963e04e018ba397151c05dc61 (127.9.176.195:45545)
I20260808 22:21:56.087229 13240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:56.083708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.193:53943 (local address 127.9.176.194:36839)
0808 22:21:56.083837 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0808 22:21:56.083842 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:21:56.084040 (+ 198us) server_negotiation.cc:407] Connection header received
0808 22:21:56.084175 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:21:56.084179 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:21:56.084229 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:21:56.084305 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:21:56.084729 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:56.085274 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:56.086327 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:56.086714 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:56.086816 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:56.086869 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0808 22:21:56.087055 (+ 186us) server_negotiation.cc:299] Negotiation successful
0808 22:21:56.087122 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 22:21:56.087244 13277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:56.083597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.194:36839 (local address 127.9.176.193:53943)
0808 22:21:56.083937 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0808 22:21:56.083952 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 22:21:56.084058 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:21:56.084335 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:21:56.084338 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:21:56.084353 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:21:56.084606 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:21:56.084615 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:21:56.085385 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:21:56.085388 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:21:56.086198 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:21:56.086207 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:21:56.086845 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:21:56.086849 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:21:56.086940 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:56.086959 (+ 19us) client_negotiation.cc:769] Sending connection context
0808 22:21:56.087022 (+ 63us) client_negotiation.cc:240] Negotiation successful
0808 22:21:56.087100 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":99,"threads_started":1}
I20260808 22:21:56.087368 13243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:56.083729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.193:37495 (local address 127.9.176.195:45545)
0808 22:21:56.083845 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0808 22:21:56.083850 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:21:56.084043 (+ 193us) server_negotiation.cc:407] Connection header received
0808 22:21:56.084277 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:21:56.084281 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:21:56.084334 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:21:56.084403 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:21:56.084959 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:56.085490 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:56.086459 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:56.086856 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:56.086959 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:56.087007 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0808 22:21:56.087180 (+ 173us) server_negotiation.cc:299] Negotiation successful
0808 22:21:56.087250 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260808 22:21:56.087545 13274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:56.083629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.195:45545 (local address 127.9.176.193:37495)
0808 22:21:56.083964 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0808 22:21:56.083975 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 22:21:56.084128 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:21:56.084430 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:21:56.084433 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:21:56.084447 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:21:56.084843 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:21:56.084850 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:21:56.085599 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:21:56.085602 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:21:56.086339 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:21:56.086348 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:21:56.086964 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:21:56.086968 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:21:56.087070 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:56.087087 (+ 17us) client_negotiation.cc:769] Sending connection context
0808 22:21:56.087153 (+ 66us) client_negotiation.cc:240] Negotiation successful
0808 22:21:56.087218 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":88,"threads_started":1}
I20260808 22:21:56.087750 12928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7b59f47cc0648149772dee7db2eb681" candidate_uuid: "ec50da59889f4a1e84a835ff268b75e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcba8d834ed44491a94b32067ba681ad" is_pre_election: true
I20260808 22:21:56.087888 12928 raft_consensus.cc:2454] T e7b59f47cc0648149772dee7db2eb681 P bcba8d834ed44491a94b32067ba681ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec50da59889f4a1e84a835ff268b75e8 in term 0.
I20260808 22:21:56.088106 13144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7b59f47cc0648149772dee7db2eb681" candidate_uuid: "ec50da59889f4a1e84a835ff268b75e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9c4150963e04e018ba397151c05dc61" is_pre_election: true
I20260808 22:21:56.088220 12609 leader_election.cc:302] T e7b59f47cc0648149772dee7db2eb681 P ec50da59889f4a1e84a835ff268b75e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bcba8d834ed44491a94b32067ba681ad, ec50da59889f4a1e84a835ff268b75e8; no voters:
I20260808 22:21:56.088299 13144 raft_consensus.cc:2454] T e7b59f47cc0648149772dee7db2eb681 P b9c4150963e04e018ba397151c05dc61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec50da59889f4a1e84a835ff268b75e8 in term 0.
I20260808 22:21:56.088496 13251 raft_consensus.cc:2781] T e7b59f47cc0648149772dee7db2eb681 P ec50da59889f4a1e84a835ff268b75e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 22:21:56.088564 13251 raft_consensus.cc:482] T e7b59f47cc0648149772dee7db2eb681 P ec50da59889f4a1e84a835ff268b75e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 22:21:56.088609 13251 raft_consensus.cc:3037] T e7b59f47cc0648149772dee7db2eb681 P ec50da59889f4a1e84a835ff268b75e8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:21:56.089586 13251 raft_consensus.cc:504] T e7b59f47cc0648149772dee7db2eb681 P ec50da59889f4a1e84a835ff268b75e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec50da59889f4a1e84a835ff268b75e8" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34469 } } peers { permanent_uuid: "bcba8d834ed44491a94b32067ba681ad" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36839 } } peers { permanent_uuid: "b9c4150963e04e018ba397151c05dc61" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45545 } }
I20260808 22:21:56.089887 13251 leader_election.cc:288] T e7b59f47cc0648149772dee7db2eb681 P ec50da59889f4a1e84a835ff268b75e8 [CANDIDATE]: Term 1 election: Requested vote from peers bcba8d834ed44491a94b32067ba681ad (127.9.176.194:36839), b9c4150963e04e018ba397151c05dc61 (127.9.176.195:45545)
I20260808 22:21:56.090226 13144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7b59f47cc0648149772dee7db2eb681" candidate_uuid: "ec50da59889f4a1e84a835ff268b75e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9c4150963e04e018ba397151c05dc61"
I20260808 22:21:56.090226 12928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7b59f47cc0648149772dee7db2eb681" candidate_uuid: "ec50da59889f4a1e84a835ff268b75e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcba8d834ed44491a94b32067ba681ad"
I20260808 22:21:56.090340 13144 raft_consensus.cc:3037] T e7b59f47cc0648149772dee7db2eb681 P b9c4150963e04e018ba397151c05dc61 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:21:56.090340 12928 raft_consensus.cc:3037] T e7b59f47cc0648149772dee7db2eb681 P bcba8d834ed44491a94b32067ba681ad [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:21:56.091405 12928 raft_consensus.cc:2454] T e7b59f47cc0648149772dee7db2eb681 P bcba8d834ed44491a94b32067ba681ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec50da59889f4a1e84a835ff268b75e8 in term 1.
I20260808 22:21:56.091405 13144 raft_consensus.cc:2454] T e7b59f47cc0648149772dee7db2eb681 P b9c4150963e04e018ba397151c05dc61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec50da59889f4a1e84a835ff268b75e8 in term 1.
I20260808 22:21:56.091698 12610 leader_election.cc:302] T e7b59f47cc0648149772dee7db2eb681 P ec50da59889f4a1e84a835ff268b75e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: b9c4150963e04e018ba397151c05dc61, bcba8d834ed44491a94b32067ba681ad, ec50da59889f4a1e84a835ff268b75e8; no voters:
I20260808 22:21:56.091863 13251 raft_consensus.cc:2781] T e7b59f47cc0648149772dee7db2eb681 P ec50da59889f4a1e84a835ff268b75e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:21:56.092068 13251 raft_consensus.cc:686] T e7b59f47cc0648149772dee7db2eb681 P ec50da59889f4a1e84a835ff268b75e8 [term 1 LEADER]: Becoming Leader. State: Replica: ec50da59889f4a1e84a835ff268b75e8, State: Running, Role: LEADER
I20260808 22:21:56.092370 13251 consensus_queue.cc:237] T e7b59f47cc0648149772dee7db2eb681 P ec50da59889f4a1e84a835ff268b75e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec50da59889f4a1e84a835ff268b75e8" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34469 } } peers { permanent_uuid: "bcba8d834ed44491a94b32067ba681ad" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36839 } } peers { permanent_uuid: "b9c4150963e04e018ba397151c05dc61" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45545 } }
I20260808 22:21:56.094694 13278 tablet_replica.cc:439] T e7b59f47cc0648149772dee7db2eb681 P ec50da59889f4a1e84a835ff268b75e8: TxnStatusTablet state changed. Reason: New leader ec50da59889f4a1e84a835ff268b75e8. Latest consensus state: current_term: 1 leader_uuid: "ec50da59889f4a1e84a835ff268b75e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec50da59889f4a1e84a835ff268b75e8" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34469 } } peers { permanent_uuid: "bcba8d834ed44491a94b32067ba681ad" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36839 } } peers { permanent_uuid: "b9c4150963e04e018ba397151c05dc61" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45545 } } }
I20260808 22:21:56.094801 13278 tablet_replica.cc:442] T e7b59f47cc0648149772dee7db2eb681 P ec50da59889f4a1e84a835ff268b75e8: This TxnStatusTablet replica's current role is: LEADER
I20260808 22:21:56.095079 13280 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260808 22:21:56.095328 13280 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260808 22:21:56.095691 12465 catalog_manager.cc:5495] T e7b59f47cc0648149772dee7db2eb681 P ec50da59889f4a1e84a835ff268b75e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec50da59889f4a1e84a835ff268b75e8 (127.9.176.193). New cstate: current_term: 1 leader_uuid: "ec50da59889f4a1e84a835ff268b75e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec50da59889f4a1e84a835ff268b75e8" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcba8d834ed44491a94b32067ba681ad" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36839 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9c4150963e04e018ba397151c05dc61" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45545 } health_report { overall_health: UNKNOWN } } }
I20260808 22:21:56.121723 13267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:56.118071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.193:34469 (local address 127.0.0.1:45450)
0808 22:21:56.118231 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0808 22:21:56.118246 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 22:21:56.118330 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:21:56.118722 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:21:56.118726 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:21:56.118734 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 22:21:56.118844 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:21:56.118850 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:21:56.119634 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:21:56.119637 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:21:56.120227 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:21:56.120234 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:21:56.120666 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:21:56.120670 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:21:56.120753 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:56.120795 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 22:21:56.121437 (+ 642us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 22:21:56.121453 (+ 16us) client_negotiation.cc:769] Sending connection context
0808 22:21:56.121519 (+ 66us) client_negotiation.cc:240] Negotiation successful
0808 22:21:56.121601 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260808 22:21:56.121721 13242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:56.118168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45450 (local address 127.9.176.193:34469)
0808 22:21:56.118485 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0808 22:21:56.118488 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:21:56.118497 (+ 9us) server_negotiation.cc:407] Connection header received
0808 22:21:56.118551 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:21:56.118553 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:21:56.118598 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:21:56.118678 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:21:56.118938 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:56.119493 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:56.120339 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:56.120557 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:56.120657 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:56.120952 (+ 295us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:21:56.121342 (+ 390us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:21:56.121444 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0808 22:21:56.121537 (+ 93us) server_negotiation.cc:299] Negotiation successful
0808 22:21:56.121587 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236}
W20260808 22:21:56.124985 13000 tablet.cc:2369] T 319c06acd55a4d6bbde8caf328e8c21a P bcba8d834ed44491a94b32067ba681ad: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 22:21:56.126842 12928 raft_consensus.cc:1261] T e7b59f47cc0648149772dee7db2eb681 P bcba8d834ed44491a94b32067ba681ad [term 1 FOLLOWER]: Refusing update from remote peer ec50da59889f4a1e84a835ff268b75e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:21:56.126895 13144 raft_consensus.cc:1261] T e7b59f47cc0648149772dee7db2eb681 P b9c4150963e04e018ba397151c05dc61 [term 1 FOLLOWER]: Refusing update from remote peer ec50da59889f4a1e84a835ff268b75e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:21:56.127472 13278 consensus_queue.cc:1035] T e7b59f47cc0648149772dee7db2eb681 P ec50da59889f4a1e84a835ff268b75e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9c4150963e04e018ba397151c05dc61" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:21:56.127632 13251 consensus_queue.cc:1035] T e7b59f47cc0648149772dee7db2eb681 P ec50da59889f4a1e84a835ff268b75e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcba8d834ed44491a94b32067ba681ad" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:21:56.130537 13255 tablet_replica.cc:439] T e7b59f47cc0648149772dee7db2eb681 P b9c4150963e04e018ba397151c05dc61: TxnStatusTablet state changed. Reason: New leader ec50da59889f4a1e84a835ff268b75e8. Latest consensus state: current_term: 1 leader_uuid: "ec50da59889f4a1e84a835ff268b75e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec50da59889f4a1e84a835ff268b75e8" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34469 } } peers { permanent_uuid: "bcba8d834ed44491a94b32067ba681ad" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36839 } } peers { permanent_uuid: "b9c4150963e04e018ba397151c05dc61" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45545 } } }
I20260808 22:21:56.130666 13255 tablet_replica.cc:442] T e7b59f47cc0648149772dee7db2eb681 P b9c4150963e04e018ba397151c05dc61: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 22:21:56.134581 13253 tablet_replica.cc:439] T e7b59f47cc0648149772dee7db2eb681 P bcba8d834ed44491a94b32067ba681ad: TxnStatusTablet state changed. Reason: New leader ec50da59889f4a1e84a835ff268b75e8. Latest consensus state: current_term: 1 leader_uuid: "ec50da59889f4a1e84a835ff268b75e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec50da59889f4a1e84a835ff268b75e8" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34469 } } peers { permanent_uuid: "bcba8d834ed44491a94b32067ba681ad" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36839 } } peers { permanent_uuid: "b9c4150963e04e018ba397151c05dc61" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45545 } } }
I20260808 22:21:56.134701 13253 tablet_replica.cc:442] T e7b59f47cc0648149772dee7db2eb681 P bcba8d834ed44491a94b32067ba681ad: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 22:21:56.134954 13251 tablet_replica.cc:439] T e7b59f47cc0648149772dee7db2eb681 P ec50da59889f4a1e84a835ff268b75e8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ec50da59889f4a1e84a835ff268b75e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec50da59889f4a1e84a835ff268b75e8" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34469 } } peers { permanent_uuid: "bcba8d834ed44491a94b32067ba681ad" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36839 } } peers { permanent_uuid: "b9c4150963e04e018ba397151c05dc61" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45545 } } }
I20260808 22:21:56.135064 13251 tablet_replica.cc:442] T e7b59f47cc0648149772dee7db2eb681 P ec50da59889f4a1e84a835ff268b75e8: This TxnStatusTablet replica's current role is: LEADER
I20260808 22:21:56.135881 13251 tablet_replica.cc:439] T e7b59f47cc0648149772dee7db2eb681 P ec50da59889f4a1e84a835ff268b75e8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ec50da59889f4a1e84a835ff268b75e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec50da59889f4a1e84a835ff268b75e8" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34469 } } peers { permanent_uuid: "bcba8d834ed44491a94b32067ba681ad" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36839 } } peers { permanent_uuid: "b9c4150963e04e018ba397151c05dc61" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45545 } } }
I20260808 22:21:56.135978 13251 tablet_replica.cc:442] T e7b59f47cc0648149772dee7db2eb681 P ec50da59889f4a1e84a835ff268b75e8: This TxnStatusTablet replica's current role is: LEADER
I20260808 22:21:56.136673 13255 tablet_replica.cc:439] T e7b59f47cc0648149772dee7db2eb681 P b9c4150963e04e018ba397151c05dc61: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ec50da59889f4a1e84a835ff268b75e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec50da59889f4a1e84a835ff268b75e8" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34469 } } peers { permanent_uuid: "bcba8d834ed44491a94b32067ba681ad" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36839 } } peers { permanent_uuid: "b9c4150963e04e018ba397151c05dc61" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45545 } } }
I20260808 22:21:56.136781 13255 tablet_replica.cc:442] T e7b59f47cc0648149772dee7db2eb681 P b9c4150963e04e018ba397151c05dc61: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 22:21:56.136858 13285 mvcc.cc:205] Tried to move back new op lower bound from 7316388725249200128 to 7316388725117788160. Current Snapshot: MvccSnapshot[applied={T|T < 7316388725249200128}]
I20260808 22:21:56.137794 13253 tablet_replica.cc:439] T e7b59f47cc0648149772dee7db2eb681 P bcba8d834ed44491a94b32067ba681ad: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ec50da59889f4a1e84a835ff268b75e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec50da59889f4a1e84a835ff268b75e8" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34469 } } peers { permanent_uuid: "bcba8d834ed44491a94b32067ba681ad" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36839 } } peers { permanent_uuid: "b9c4150963e04e018ba397151c05dc61" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45545 } } }
I20260808 22:21:56.137897 13253 tablet_replica.cc:442] T e7b59f47cc0648149772dee7db2eb681 P bcba8d834ed44491a94b32067ba681ad: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 22:21:56.138202 13288 mvcc.cc:205] Tried to move back new op lower bound from 7316388725249200128 to 7316388725117788160. Current Snapshot: MvccSnapshot[applied={T|T < 7316388725249200128}]
I20260808 22:21:56.138579 13284 mvcc.cc:205] Tried to move back new op lower bound from 7316388725249200128 to 7316388725117788160. Current Snapshot: MvccSnapshot[applied={T|T < 7316388725249200128}]
I20260808 22:21:56.148082 13243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:56.144728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50908 (local address 127.9.176.195:45545)
0808 22:21:56.144836 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0808 22:21:56.144840 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:21:56.144925 (+ 85us) server_negotiation.cc:407] Connection header received
0808 22:21:56.145094 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:21:56.145097 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:21:56.145154 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:21:56.145235 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:21:56.145477 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:56.145971 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:56.146809 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:56.147023 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:56.147109 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:56.147359 (+ 250us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:21:56.147723 (+ 364us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:21:56.147821 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0808 22:21:56.147909 (+ 88us) server_negotiation.cc:299] Negotiation successful
0808 22:21:56.147962 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 22:21:56.157750 13218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:56.154632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.193:34469 (local address 127.9.176.195:33999)
0808 22:21:56.154839 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0808 22:21:56.154853 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 22:21:56.154939 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:21:56.155165 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:21:56.155167 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:21:56.155174 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 22:21:56.155272 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:21:56.155278 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:21:56.155948 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:21:56.155952 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:21:56.156539 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:21:56.156547 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:21:56.157024 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:21:56.157028 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:21:56.157111 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:56.157159 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 22:21:56.157484 (+ 325us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 22:21:56.157492 (+ 8us) client_negotiation.cc:769] Sending connection context
0808 22:21:56.157553 (+ 61us) client_negotiation.cc:240] Negotiation successful
0808 22:21:56.157646 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117}
I20260808 22:21:56.157747 13242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:56.154731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.195:33999 (local address 127.9.176.193:34469)
0808 22:21:56.154846 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0808 22:21:56.154851 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:21:56.154894 (+ 43us) server_negotiation.cc:407] Connection header received
0808 22:21:56.155040 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:21:56.155043 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:21:56.155085 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:21:56.155141 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:21:56.155372 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:56.155849 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:56.156657 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:56.156866 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:56.156973 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:56.157301 (+ 328us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:21:56.157380 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:21:56.157441 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0808 22:21:56.157581 (+ 140us) server_negotiation.cc:299] Negotiation successful
0808 22:21:56.157627 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 22:21:56.165014 12928 raft_consensus.cc:1261] T 319c06acd55a4d6bbde8caf328e8c21a P bcba8d834ed44491a94b32067ba681ad [term 1 FOLLOWER]: Refusing update from remote peer b9c4150963e04e018ba397151c05dc61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:21:56.165191 12716 raft_consensus.cc:1261] T 319c06acd55a4d6bbde8caf328e8c21a P ec50da59889f4a1e84a835ff268b75e8 [term 1 FOLLOWER]: Refusing update from remote peer b9c4150963e04e018ba397151c05dc61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:21:56.165694 13258 consensus_queue.cc:1035] T 319c06acd55a4d6bbde8caf328e8c21a P b9c4150963e04e018ba397151c05dc61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec50da59889f4a1e84a835ff268b75e8" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:21:56.166069 13258 consensus_queue.cc:1035] T 319c06acd55a4d6bbde8caf328e8c21a P b9c4150963e04e018ba397151c05dc61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcba8d834ed44491a94b32067ba681ad" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:21:56.189818 13243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:56.186022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.193:40041 (local address 127.9.176.195:45545)
0808 22:21:56.186147 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0808 22:21:56.186152 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:21:56.186355 (+ 203us) server_negotiation.cc:407] Connection header received
0808 22:21:56.186448 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:21:56.186451 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:21:56.186499 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:21:56.186575 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:21:56.186770 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:56.187550 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:56.188765 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:56.189042 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:56.189124 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:56.189403 (+ 279us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:21:56.189467 (+ 64us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:21:56.189515 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0808 22:21:56.189626 (+ 111us) server_negotiation.cc:299] Negotiation successful
0808 22:21:56.189673 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 22:21:56.189839 13308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:56.185986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.195:45545 (local address 127.9.176.193:40041)
0808 22:21:56.186278 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0808 22:21:56.186292 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 22:21:56.186377 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:21:56.186579 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:21:56.186581 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:21:56.186587 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 22:21:56.186684 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:21:56.186689 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:21:56.187681 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:21:56.187685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:21:56.188650 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:21:56.188661 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:21:56.189140 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:21:56.189143 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:21:56.189235 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:56.189289 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 22:21:56.189540 (+ 251us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 22:21:56.189546 (+ 6us) client_negotiation.cc:769] Sending connection context
0808 22:21:56.189602 (+ 56us) client_negotiation.cc:240] Negotiation successful
0808 22:21:56.189682 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":109,"threads_started":1}
I20260808 22:21:56.215757 12464 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40622:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:21:56.215962 12464 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40622:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:21:56.219558 12464 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a9a81a719cd649ec800f8a3cc702574d: Sending DeleteTablet for 3 replicas of tablet 319c06acd55a4d6bbde8caf328e8c21a
I20260808 22:21:56.220391 12904 tablet_service.cc:1514] Processing DeleteTablet for tablet 319c06acd55a4d6bbde8caf328e8c21a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:21:56 UTC) from {username='kudu', principal='kudu/127.9.176.254@KRBTEST.COM'} at 127.0.0.1:46780
I20260808 22:21:56.220558 13113 tablet_service.cc:1514] Processing DeleteTablet for tablet 319c06acd55a4d6bbde8caf328e8c21a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:21:56 UTC) from {username='kudu', principal='kudu/127.9.176.254@KRBTEST.COM'} at 127.0.0.1:50896
I20260808 22:21:56.221117 13314 tablet_replica.cc:330] T 319c06acd55a4d6bbde8caf328e8c21a P b9c4150963e04e018ba397151c05dc61: stopping tablet replica
I20260808 22:21:56.221426 13314 raft_consensus.cc:2229] T 319c06acd55a4d6bbde8caf328e8c21a P b9c4150963e04e018ba397151c05dc61 [term 1 LEADER]: Raft consensus shutting down.
I20260808 22:21:56.221629 12681 tablet_service.cc:1514] Processing DeleteTablet for tablet 319c06acd55a4d6bbde8caf328e8c21a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:21:56 UTC) from {username='kudu', principal='kudu/127.9.176.254@KRBTEST.COM'} at 127.0.0.1:45434
I20260808 22:21:56.221818 13314 raft_consensus.cc:2258] T 319c06acd55a4d6bbde8caf328e8c21a P b9c4150963e04e018ba397151c05dc61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:21:56.222932 12464 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40622:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 22:21:56.223368 12464 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 22:21:56.223474 13314 ts_tablet_manager.cc:1905] T 319c06acd55a4d6bbde8caf328e8c21a P b9c4150963e04e018ba397151c05dc61: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:21:56.225318 13315 tablet_replica.cc:330] T 319c06acd55a4d6bbde8caf328e8c21a P ec50da59889f4a1e84a835ff268b75e8: stopping tablet replica
I20260808 22:21:56.225692 13315 raft_consensus.cc:2229] T 319c06acd55a4d6bbde8caf328e8c21a P ec50da59889f4a1e84a835ff268b75e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:21:56.225906 13315 raft_consensus.cc:2258] T 319c06acd55a4d6bbde8caf328e8c21a P ec50da59889f4a1e84a835ff268b75e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:21:56.227811 13314 ts_tablet_manager.cc:1918] T 319c06acd55a4d6bbde8caf328e8c21a P b9c4150963e04e018ba397151c05dc61: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 22:21:56.227917 13314 log.cc:1196] T 319c06acd55a4d6bbde8caf328e8c21a P b9c4150963e04e018ba397151c05dc61: Deleting WAL directory at /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/minicluster-data/ts-2/wal/wals/319c06acd55a4d6bbde8caf328e8c21a
I20260808 22:21:56.228253 13314 ts_tablet_manager.cc:1939] T 319c06acd55a4d6bbde8caf328e8c21a P b9c4150963e04e018ba397151c05dc61: Deleting consensus metadata
I20260808 22:21:56.228533 13315 ts_tablet_manager.cc:1905] T 319c06acd55a4d6bbde8caf328e8c21a P ec50da59889f4a1e84a835ff268b75e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:21:56.229202 13113 tablet_service.cc:1467] Processing CreateTablet for tablet 887394abbe8d4262b416db2830805e84 (DEFAULT_TABLE table=test-table [id=a8a0bd79ef344148b7d970bd8d25b537]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:21:56.229358 12681 tablet_service.cc:1467] Processing CreateTablet for tablet 887394abbe8d4262b416db2830805e84 (DEFAULT_TABLE table=test-table [id=a8a0bd79ef344148b7d970bd8d25b537]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:21:56.229406 12904 tablet_service.cc:1467] Processing CreateTablet for tablet 887394abbe8d4262b416db2830805e84 (DEFAULT_TABLE table=test-table [id=a8a0bd79ef344148b7d970bd8d25b537]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:21:56.229458 13113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 887394abbe8d4262b416db2830805e84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:21:56.229691 12904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 887394abbe8d4262b416db2830805e84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:21:56.229820 13312 tablet_replica.cc:330] T 319c06acd55a4d6bbde8caf328e8c21a P bcba8d834ed44491a94b32067ba681ad: stopping tablet replica
I20260808 22:21:56.230543 13312 raft_consensus.cc:2229] T 319c06acd55a4d6bbde8caf328e8c21a P bcba8d834ed44491a94b32067ba681ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:21:56.231299 13312 raft_consensus.cc:2258] T 319c06acd55a4d6bbde8caf328e8c21a P bcba8d834ed44491a94b32067ba681ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:21:56.231796 13246 tablet_bootstrap.cc:492] T 887394abbe8d4262b416db2830805e84 P b9c4150963e04e018ba397151c05dc61: Bootstrap starting.
I20260808 22:21:56.232132 13244 tablet_bootstrap.cc:492] T 887394abbe8d4262b416db2830805e84 P bcba8d834ed44491a94b32067ba681ad: Bootstrap starting.
I20260808 22:21:56.232720 13246 tablet_bootstrap.cc:654] T 887394abbe8d4262b416db2830805e84 P b9c4150963e04e018ba397151c05dc61: Neither blocks nor log segments found. Creating new log.
I20260808 22:21:56.234313 13244 tablet_bootstrap.cc:654] T 887394abbe8d4262b416db2830805e84 P bcba8d834ed44491a94b32067ba681ad: Neither blocks nor log segments found. Creating new log.
I20260808 22:21:56.234270 12449 catalog_manager.cc:4841] TS b9c4150963e04e018ba397151c05dc61 (127.9.176.195:45545): tablet 319c06acd55a4d6bbde8caf328e8c21a (table test-table [id=9fb1111cbf154052bf145668cd6fdb93]) successfully deleted
I20260808 22:21:56.235167 13246 tablet_bootstrap.cc:492] T 887394abbe8d4262b416db2830805e84 P b9c4150963e04e018ba397151c05dc61: No bootstrap required, opened a new log
I20260808 22:21:56.235265 13246 ts_tablet_manager.cc:1397] T 887394abbe8d4262b416db2830805e84 P b9c4150963e04e018ba397151c05dc61: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260808 22:21:56.235726 13246 raft_consensus.cc:348] T 887394abbe8d4262b416db2830805e84 P b9c4150963e04e018ba397151c05dc61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec50da59889f4a1e84a835ff268b75e8" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34469 } } peers { permanent_uuid: "bcba8d834ed44491a94b32067ba681ad" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36839 } } peers { permanent_uuid: "b9c4150963e04e018ba397151c05dc61" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45545 } }
I20260808 22:21:56.235852 13246 raft_consensus.cc:374] T 887394abbe8d4262b416db2830805e84 P b9c4150963e04e018ba397151c05dc61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:21:56.235890 13246 raft_consensus.cc:729] T 887394abbe8d4262b416db2830805e84 P b9c4150963e04e018ba397151c05dc61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9c4150963e04e018ba397151c05dc61, State: Initialized, Role: FOLLOWER
I20260808 22:21:56.236003 13246 consensus_queue.cc:260] T 887394abbe8d4262b416db2830805e84 P b9c4150963e04e018ba397151c05dc61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec50da59889f4a1e84a835ff268b75e8" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34469 } } peers { permanent_uuid: "bcba8d834ed44491a94b32067ba681ad" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36839 } } peers { permanent_uuid: "b9c4150963e04e018ba397151c05dc61" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45545 } }
I20260808 22:21:56.236181 12681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 887394abbe8d4262b416db2830805e84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:21:56.236289 13246 ts_tablet_manager.cc:1428] T 887394abbe8d4262b416db2830805e84 P b9c4150963e04e018ba397151c05dc61: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260808 22:21:56.238387 13245 tablet_bootstrap.cc:492] T 887394abbe8d4262b416db2830805e84 P ec50da59889f4a1e84a835ff268b75e8: Bootstrap starting.
I20260808 22:21:56.238627 13312 ts_tablet_manager.cc:1905] T 319c06acd55a4d6bbde8caf328e8c21a P bcba8d834ed44491a94b32067ba681ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:21:56.239331 13245 tablet_bootstrap.cc:654] T 887394abbe8d4262b416db2830805e84 P ec50da59889f4a1e84a835ff268b75e8: Neither blocks nor log segments found. Creating new log.
I20260808 22:21:56.242348 13312 ts_tablet_manager.cc:1918] T 319c06acd55a4d6bbde8caf328e8c21a P bcba8d834ed44491a94b32067ba681ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 22:21:56.242491 13312 log.cc:1196] T 319c06acd55a4d6bbde8caf328e8c21a P bcba8d834ed44491a94b32067ba681ad: Deleting WAL directory at /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/minicluster-data/ts-1/wal/wals/319c06acd55a4d6bbde8caf328e8c21a
I20260808 22:21:56.242791 13315 ts_tablet_manager.cc:1918] T 319c06acd55a4d6bbde8caf328e8c21a P ec50da59889f4a1e84a835ff268b75e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 22:21:56.242894 13315 log.cc:1196] T 319c06acd55a4d6bbde8caf328e8c21a P ec50da59889f4a1e84a835ff268b75e8: Deleting WAL directory at /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/minicluster-data/ts-0/wal/wals/319c06acd55a4d6bbde8caf328e8c21a
I20260808 22:21:56.242908 13312 ts_tablet_manager.cc:1939] T 319c06acd55a4d6bbde8caf328e8c21a P bcba8d834ed44491a94b32067ba681ad: Deleting consensus metadata
I20260808 22:21:56.243250 13315 ts_tablet_manager.cc:1939] T 319c06acd55a4d6bbde8caf328e8c21a P ec50da59889f4a1e84a835ff268b75e8: Deleting consensus metadata
I20260808 22:21:56.244407 12450 catalog_manager.cc:4841] TS ec50da59889f4a1e84a835ff268b75e8 (127.9.176.193:34469): tablet 319c06acd55a4d6bbde8caf328e8c21a (table test-table [id=9fb1111cbf154052bf145668cd6fdb93]) successfully deleted
I20260808 22:21:56.244939 12448 catalog_manager.cc:4841] TS bcba8d834ed44491a94b32067ba681ad (127.9.176.194:36839): tablet 319c06acd55a4d6bbde8caf328e8c21a (table test-table [id=9fb1111cbf154052bf145668cd6fdb93]) successfully deleted
I20260808 22:21:56.245002 13244 tablet_bootstrap.cc:492] T 887394abbe8d4262b416db2830805e84 P bcba8d834ed44491a94b32067ba681ad: No bootstrap required, opened a new log
I20260808 22:21:56.245095 13244 ts_tablet_manager.cc:1397] T 887394abbe8d4262b416db2830805e84 P bcba8d834ed44491a94b32067ba681ad: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.001s
I20260808 22:21:56.245561 13244 raft_consensus.cc:348] T 887394abbe8d4262b416db2830805e84 P bcba8d834ed44491a94b32067ba681ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec50da59889f4a1e84a835ff268b75e8" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34469 } } peers { permanent_uuid: "bcba8d834ed44491a94b32067ba681ad" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36839 } } peers { permanent_uuid: "b9c4150963e04e018ba397151c05dc61" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45545 } }
I20260808 22:21:56.245687 13244 raft_consensus.cc:374] T 887394abbe8d4262b416db2830805e84 P bcba8d834ed44491a94b32067ba681ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:21:56.245728 13244 raft_consensus.cc:729] T 887394abbe8d4262b416db2830805e84 P bcba8d834ed44491a94b32067ba681ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcba8d834ed44491a94b32067ba681ad, State: Initialized, Role: FOLLOWER
I20260808 22:21:56.245858 13244 consensus_queue.cc:260] T 887394abbe8d4262b416db2830805e84 P bcba8d834ed44491a94b32067ba681ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec50da59889f4a1e84a835ff268b75e8" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34469 } } peers { permanent_uuid: "bcba8d834ed44491a94b32067ba681ad" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36839 } } peers { permanent_uuid: "b9c4150963e04e018ba397151c05dc61" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45545 } }
I20260808 22:21:56.246140 13244 ts_tablet_manager.cc:1428] T 887394abbe8d4262b416db2830805e84 P bcba8d834ed44491a94b32067ba681ad: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260808 22:21:56.246940 13245 tablet_bootstrap.cc:492] T 887394abbe8d4262b416db2830805e84 P ec50da59889f4a1e84a835ff268b75e8: No bootstrap required, opened a new log
I20260808 22:21:56.247026 13245 ts_tablet_manager.cc:1397] T 887394abbe8d4262b416db2830805e84 P ec50da59889f4a1e84a835ff268b75e8: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260808 22:21:56.247460 13245 raft_consensus.cc:348] T 887394abbe8d4262b416db2830805e84 P ec50da59889f4a1e84a835ff268b75e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec50da59889f4a1e84a835ff268b75e8" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34469 } } peers { permanent_uuid: "bcba8d834ed44491a94b32067ba681ad" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36839 } } peers { permanent_uuid: "b9c4150963e04e018ba397151c05dc61" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45545 } }
I20260808 22:21:56.247572 13245 raft_consensus.cc:374] T 887394abbe8d4262b416db2830805e84 P ec50da59889f4a1e84a835ff268b75e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:21:56.247614 13245 raft_consensus.cc:729] T 887394abbe8d4262b416db2830805e84 P ec50da59889f4a1e84a835ff268b75e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec50da59889f4a1e84a835ff268b75e8, State: Initialized, Role: FOLLOWER
I20260808 22:21:56.247735 13245 consensus_queue.cc:260] T 887394abbe8d4262b416db2830805e84 P ec50da59889f4a1e84a835ff268b75e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec50da59889f4a1e84a835ff268b75e8" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34469 } } peers { permanent_uuid: "bcba8d834ed44491a94b32067ba681ad" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36839 } } peers { permanent_uuid: "b9c4150963e04e018ba397151c05dc61" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45545 } }
I20260808 22:21:56.247995 13245 ts_tablet_manager.cc:1428] T 887394abbe8d4262b416db2830805e84 P ec50da59889f4a1e84a835ff268b75e8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260808 22:21:56.482614 13278 raft_consensus.cc:482] T 887394abbe8d4262b416db2830805e84 P ec50da59889f4a1e84a835ff268b75e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 22:21:56.482774 13278 raft_consensus.cc:504] T 887394abbe8d4262b416db2830805e84 P ec50da59889f4a1e84a835ff268b75e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec50da59889f4a1e84a835ff268b75e8" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34469 } } peers { permanent_uuid: "bcba8d834ed44491a94b32067ba681ad" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36839 } } peers { permanent_uuid: "b9c4150963e04e018ba397151c05dc61" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45545 } }
I20260808 22:21:56.483186 13278 leader_election.cc:288] T 887394abbe8d4262b416db2830805e84 P ec50da59889f4a1e84a835ff268b75e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bcba8d834ed44491a94b32067ba681ad (127.9.176.194:36839), b9c4150963e04e018ba397151c05dc61 (127.9.176.195:45545)
I20260808 22:21:56.483680 13144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "887394abbe8d4262b416db2830805e84" candidate_uuid: "ec50da59889f4a1e84a835ff268b75e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9c4150963e04e018ba397151c05dc61" is_pre_election: true
I20260808 22:21:56.483708 12928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "887394abbe8d4262b416db2830805e84" candidate_uuid: "ec50da59889f4a1e84a835ff268b75e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcba8d834ed44491a94b32067ba681ad" is_pre_election: true
I20260808 22:21:56.483839 12928 raft_consensus.cc:2454] T 887394abbe8d4262b416db2830805e84 P bcba8d834ed44491a94b32067ba681ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec50da59889f4a1e84a835ff268b75e8 in term 0.
I20260808 22:21:56.483848 13144 raft_consensus.cc:2454] T 887394abbe8d4262b416db2830805e84 P b9c4150963e04e018ba397151c05dc61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec50da59889f4a1e84a835ff268b75e8 in term 0.
I20260808 22:21:56.484158 12609 leader_election.cc:302] T 887394abbe8d4262b416db2830805e84 P ec50da59889f4a1e84a835ff268b75e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bcba8d834ed44491a94b32067ba681ad, ec50da59889f4a1e84a835ff268b75e8; no voters:
I20260808 22:21:56.484339 13278 raft_consensus.cc:2781] T 887394abbe8d4262b416db2830805e84 P ec50da59889f4a1e84a835ff268b75e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 22:21:56.484390 13278 raft_consensus.cc:482] T 887394abbe8d4262b416db2830805e84 P ec50da59889f4a1e84a835ff268b75e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 22:21:56.484416 13278 raft_consensus.cc:3037] T 887394abbe8d4262b416db2830805e84 P ec50da59889f4a1e84a835ff268b75e8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:21:56.485495 13278 raft_consensus.cc:504] T 887394abbe8d4262b416db2830805e84 P ec50da59889f4a1e84a835ff268b75e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec50da59889f4a1e84a835ff268b75e8" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34469 } } peers { permanent_uuid: "bcba8d834ed44491a94b32067ba681ad" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36839 } } peers { permanent_uuid: "b9c4150963e04e018ba397151c05dc61" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45545 } }
I20260808 22:21:56.485812 13278 leader_election.cc:288] T 887394abbe8d4262b416db2830805e84 P ec50da59889f4a1e84a835ff268b75e8 [CANDIDATE]: Term 1 election: Requested vote from peers bcba8d834ed44491a94b32067ba681ad (127.9.176.194:36839), b9c4150963e04e018ba397151c05dc61 (127.9.176.195:45545)
I20260808 22:21:56.486166 12928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "887394abbe8d4262b416db2830805e84" candidate_uuid: "ec50da59889f4a1e84a835ff268b75e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcba8d834ed44491a94b32067ba681ad"
I20260808 22:21:56.486181 13144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "887394abbe8d4262b416db2830805e84" candidate_uuid: "ec50da59889f4a1e84a835ff268b75e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9c4150963e04e018ba397151c05dc61"
I20260808 22:21:56.486280 12928 raft_consensus.cc:3037] T 887394abbe8d4262b416db2830805e84 P bcba8d834ed44491a94b32067ba681ad [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:21:56.486281 13144 raft_consensus.cc:3037] T 887394abbe8d4262b416db2830805e84 P b9c4150963e04e018ba397151c05dc61 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:21:56.487165 13144 raft_consensus.cc:2454] T 887394abbe8d4262b416db2830805e84 P b9c4150963e04e018ba397151c05dc61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec50da59889f4a1e84a835ff268b75e8 in term 1.
I20260808 22:21:56.487269 12928 raft_consensus.cc:2454] T 887394abbe8d4262b416db2830805e84 P bcba8d834ed44491a94b32067ba681ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec50da59889f4a1e84a835ff268b75e8 in term 1.
I20260808 22:21:56.487423 12610 leader_election.cc:302] T 887394abbe8d4262b416db2830805e84 P ec50da59889f4a1e84a835ff268b75e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9c4150963e04e018ba397151c05dc61, ec50da59889f4a1e84a835ff268b75e8; no voters:
I20260808 22:21:56.487582 13278 raft_consensus.cc:2781] T 887394abbe8d4262b416db2830805e84 P ec50da59889f4a1e84a835ff268b75e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:21:56.487671 13278 raft_consensus.cc:686] T 887394abbe8d4262b416db2830805e84 P ec50da59889f4a1e84a835ff268b75e8 [term 1 LEADER]: Becoming Leader. State: Replica: ec50da59889f4a1e84a835ff268b75e8, State: Running, Role: LEADER
I20260808 22:21:56.487833 13278 consensus_queue.cc:237] T 887394abbe8d4262b416db2830805e84 P ec50da59889f4a1e84a835ff268b75e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec50da59889f4a1e84a835ff268b75e8" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34469 } } peers { permanent_uuid: "bcba8d834ed44491a94b32067ba681ad" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36839 } } peers { permanent_uuid: "b9c4150963e04e018ba397151c05dc61" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45545 } }
I20260808 22:21:56.489501 12463 catalog_manager.cc:5495] T 887394abbe8d4262b416db2830805e84 P ec50da59889f4a1e84a835ff268b75e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec50da59889f4a1e84a835ff268b75e8 (127.9.176.193). New cstate: current_term: 1 leader_uuid: "ec50da59889f4a1e84a835ff268b75e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec50da59889f4a1e84a835ff268b75e8" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcba8d834ed44491a94b32067ba681ad" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36839 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9c4150963e04e018ba397151c05dc61" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45545 } health_report { overall_health: UNKNOWN } } }
I20260808 22:21:56.533427 13242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:56.530087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45466 (local address 127.9.176.193:34469)
0808 22:21:56.530205 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0808 22:21:56.530211 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 22:21:56.530225 (+ 14us) server_negotiation.cc:407] Connection header received
0808 22:21:56.530353 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:21:56.530356 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:21:56.530411 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:21:56.530482 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:21:56.530732 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:56.531350 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:56.532284 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:56.532553 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:56.532648 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:56.532945 (+ 297us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:21:56.533024 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:21:56.533104 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0808 22:21:56.533228 (+ 124us) server_negotiation.cc:299] Negotiation successful
0808 22:21:56.533277 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260808 22:21:56.541320 13308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:56.538728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.193:34469 (local address 127.9.176.193:57601)
0808 22:21:56.538855 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0808 22:21:56.538867 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 22:21:56.538926 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:21:56.539104 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:21:56.539106 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:21:56.539111 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 22:21:56.539190 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:21:56.539194 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:21:56.539799 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:21:56.539800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:21:56.540353 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:21:56.540359 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:21:56.540751 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:21:56.540753 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:21:56.540842 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:56.540864 (+ 22us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 22:21:56.541113 (+ 249us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 22:21:56.541124 (+ 11us) client_negotiation.cc:769] Sending connection context
0808 22:21:56.541163 (+ 39us) client_negotiation.cc:240] Negotiation successful
0808 22:21:56.541222 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":26}
I20260808 22:21:56.541321 13242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:56.538796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.193:57601 (local address 127.9.176.193:34469)
0808 22:21:56.538886 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0808 22:21:56.538888 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:21:56.538900 (+ 12us) server_negotiation.cc:407] Connection header received
0808 22:21:56.538999 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:21:56.539001 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:21:56.539038 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:21:56.539095 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:21:56.539256 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:56.539730 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:56.540460 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:56.540666 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:56.540777 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:56.540994 (+ 217us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:21:56.541051 (+ 57us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:21:56.541102 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0808 22:21:56.541177 (+ 75us) server_negotiation.cc:299] Negotiation successful
0808 22:21:56.541213 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":31}
I20260808 22:21:56.546056 12928 raft_consensus.cc:1261] T 887394abbe8d4262b416db2830805e84 P bcba8d834ed44491a94b32067ba681ad [term 1 FOLLOWER]: Refusing update from remote peer ec50da59889f4a1e84a835ff268b75e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:21:56.546140 13144 raft_consensus.cc:1261] T 887394abbe8d4262b416db2830805e84 P b9c4150963e04e018ba397151c05dc61 [term 1 FOLLOWER]: Refusing update from remote peer ec50da59889f4a1e84a835ff268b75e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:21:56.546479 13251 consensus_queue.cc:1035] T 887394abbe8d4262b416db2830805e84 P ec50da59889f4a1e84a835ff268b75e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcba8d834ed44491a94b32067ba681ad" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:21:56.546615 13278 consensus_queue.cc:1035] T 887394abbe8d4262b416db2830805e84 P ec50da59889f4a1e84a835ff268b75e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9c4150963e04e018ba397151c05dc61" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 22:21:56.560284 13284 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260808 22:21:58.109798 12465 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40622:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:21:58.109953 12465 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40622:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:21:58.112139 12465 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a9a81a719cd649ec800f8a3cc702574d: Sending DeleteTablet for 3 replicas of tablet 887394abbe8d4262b416db2830805e84
I20260808 22:21:58.112609 12680 tablet_service.cc:1514] Processing DeleteTablet for tablet 887394abbe8d4262b416db2830805e84 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:21:58 UTC) from {username='kudu', principal='kudu/127.9.176.254@KRBTEST.COM'} at 127.0.0.1:45434
I20260808 22:21:58.112706 12904 tablet_service.cc:1514] Processing DeleteTablet for tablet 887394abbe8d4262b416db2830805e84 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:21:58 UTC) from {username='kudu', principal='kudu/127.9.176.254@KRBTEST.COM'} at 127.0.0.1:46780
I20260808 22:21:58.112944 13370 tablet_replica.cc:330] T 887394abbe8d4262b416db2830805e84 P ec50da59889f4a1e84a835ff268b75e8: stopping tablet replica
I20260808 22:21:58.113088 13370 raft_consensus.cc:2229] T 887394abbe8d4262b416db2830805e84 P ec50da59889f4a1e84a835ff268b75e8 [term 1 LEADER]: Raft consensus shutting down.
I20260808 22:21:58.113168 13371 tablet_replica.cc:330] T 887394abbe8d4262b416db2830805e84 P bcba8d834ed44491a94b32067ba681ad: stopping tablet replica
I20260808 22:21:58.113269 13371 raft_consensus.cc:2229] T 887394abbe8d4262b416db2830805e84 P bcba8d834ed44491a94b32067ba681ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:21:58.113356 13371 raft_consensus.cc:2258] T 887394abbe8d4262b416db2830805e84 P bcba8d834ed44491a94b32067ba681ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:21:58.113374 13370 raft_consensus.cc:2258] T 887394abbe8d4262b416db2830805e84 P ec50da59889f4a1e84a835ff268b75e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:21:58.113953 13371 ts_tablet_manager.cc:1905] T 887394abbe8d4262b416db2830805e84 P bcba8d834ed44491a94b32067ba681ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:21:58.114069 13370 ts_tablet_manager.cc:1905] T 887394abbe8d4262b416db2830805e84 P ec50da59889f4a1e84a835ff268b75e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:21:58.114871 13113 tablet_service.cc:1514] Processing DeleteTablet for tablet 887394abbe8d4262b416db2830805e84 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:21:58 UTC) from {username='kudu', principal='kudu/127.9.176.254@KRBTEST.COM'} at 127.0.0.1:50896
I20260808 22:21:58.116657 13370 ts_tablet_manager.cc:1918] T 887394abbe8d4262b416db2830805e84 P ec50da59889f4a1e84a835ff268b75e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 22:21:58.116739 13370 log.cc:1196] T 887394abbe8d4262b416db2830805e84 P ec50da59889f4a1e84a835ff268b75e8: Deleting WAL directory at /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/minicluster-data/ts-0/wal/wals/887394abbe8d4262b416db2830805e84
I20260808 22:21:58.117002 13371 ts_tablet_manager.cc:1918] T 887394abbe8d4262b416db2830805e84 P bcba8d834ed44491a94b32067ba681ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 22:21:58.117066 13370 ts_tablet_manager.cc:1939] T 887394abbe8d4262b416db2830805e84 P ec50da59889f4a1e84a835ff268b75e8: Deleting consensus metadata
I20260808 22:21:58.117081 13371 log.cc:1196] T 887394abbe8d4262b416db2830805e84 P bcba8d834ed44491a94b32067ba681ad: Deleting WAL directory at /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/minicluster-data/ts-1/wal/wals/887394abbe8d4262b416db2830805e84
I20260808 22:21:58.117297 13371 ts_tablet_manager.cc:1939] T 887394abbe8d4262b416db2830805e84 P bcba8d834ed44491a94b32067ba681ad: Deleting consensus metadata
I20260808 22:21:58.117692 12450 catalog_manager.cc:4841] TS ec50da59889f4a1e84a835ff268b75e8 (127.9.176.193:34469): tablet 887394abbe8d4262b416db2830805e84 (table test-table [id=a8a0bd79ef344148b7d970bd8d25b537]) successfully deleted
I20260808 22:21:58.117851 12448 catalog_manager.cc:4841] TS bcba8d834ed44491a94b32067ba681ad (127.9.176.194:36839): tablet 887394abbe8d4262b416db2830805e84 (table test-table [id=a8a0bd79ef344148b7d970bd8d25b537]) successfully deleted
I20260808 22:21:58.119575 13373 tablet_replica.cc:330] T 887394abbe8d4262b416db2830805e84 P b9c4150963e04e018ba397151c05dc61: stopping tablet replica
I20260808 22:21:58.119701 13373 raft_consensus.cc:2229] T 887394abbe8d4262b416db2830805e84 P b9c4150963e04e018ba397151c05dc61 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:21:58.119818 13373 raft_consensus.cc:2258] T 887394abbe8d4262b416db2830805e84 P b9c4150963e04e018ba397151c05dc61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:21:58.120383 13373 ts_tablet_manager.cc:1905] T 887394abbe8d4262b416db2830805e84 P b9c4150963e04e018ba397151c05dc61: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:21:58.122690 13373 ts_tablet_manager.cc:1918] T 887394abbe8d4262b416db2830805e84 P b9c4150963e04e018ba397151c05dc61: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 22:21:58.122751 13373 log.cc:1196] T 887394abbe8d4262b416db2830805e84 P b9c4150963e04e018ba397151c05dc61: Deleting WAL directory at /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/minicluster-data/ts-2/wal/wals/887394abbe8d4262b416db2830805e84
I20260808 22:21:58.122942 13373 ts_tablet_manager.cc:1939] T 887394abbe8d4262b416db2830805e84 P b9c4150963e04e018ba397151c05dc61: Deleting consensus metadata
I20260808 22:21:58.123543 12449 catalog_manager.cc:4841] TS b9c4150963e04e018ba397151c05dc61 (127.9.176.195:45545): tablet 887394abbe8d4262b416db2830805e84 (table test-table [id=a8a0bd79ef344148b7d970bd8d25b537]) successfully deleted
Aug 08 22:21:58 dist-test-slave-ncp9 krb5kdc[12408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227718, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 22:21:58 dist-test-slave-ncp9 krb5kdc[12408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227718, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
I20260808 22:21:58.140448 13385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:58.132630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40638 (local address 127.9.176.254:44743)
0808 22:21:58.133004 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0808 22:21:58.133011 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 22:21:58.133033 (+ 22us) server_negotiation.cc:407] Connection header received
0808 22:21:58.133109 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:21:58.133112 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:21:58.133153 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:21:58.133225 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:21:58.133825 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:58.134364 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:58.135150 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:58.135367 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:58.135438 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:58.137692 (+ 2254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:21:58.137695 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:21:58.137696 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:21:58.137720 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:21:58.139074 (+ 1354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:21:58.139440 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:21:58.139443 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:21:58.139444 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:21:58.139512 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:21:58.139694 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:21:58.139696 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:21:58.139697 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:21:58.139854 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:21:58.139945 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0808 22:21:58.140196 (+ 251us) server_negotiation.cc:299] Negotiation successful
0808 22:21:58.140288 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":131,"threads_started":1}
I20260808 22:21:58.142961 12465 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40638:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 22:21:58.143390 12465 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 22:21:58.147756 12680 tablet_service.cc:1467] Processing CreateTablet for tablet a9ad20cd392f4b168b74a92d991c4b9a (DEFAULT_TABLE table=test-table [id=790328f1386d455c8fa36b7bc0318e4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:21:58.147867 12904 tablet_service.cc:1467] Processing CreateTablet for tablet a9ad20cd392f4b168b74a92d991c4b9a (DEFAULT_TABLE table=test-table [id=790328f1386d455c8fa36b7bc0318e4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:21:58.147948 13113 tablet_service.cc:1467] Processing CreateTablet for tablet a9ad20cd392f4b168b74a92d991c4b9a (DEFAULT_TABLE table=test-table [id=790328f1386d455c8fa36b7bc0318e4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:21:58.148059 12680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9ad20cd392f4b168b74a92d991c4b9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:21:58.148166 12904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9ad20cd392f4b168b74a92d991c4b9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:21:58.148177 13113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9ad20cd392f4b168b74a92d991c4b9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:21:58.150327 13386 tablet_bootstrap.cc:492] T a9ad20cd392f4b168b74a92d991c4b9a P ec50da59889f4a1e84a835ff268b75e8: Bootstrap starting.
I20260808 22:21:58.150635 13389 tablet_bootstrap.cc:492] T a9ad20cd392f4b168b74a92d991c4b9a P b9c4150963e04e018ba397151c05dc61: Bootstrap starting.
I20260808 22:21:58.151024 13390 tablet_bootstrap.cc:492] T a9ad20cd392f4b168b74a92d991c4b9a P bcba8d834ed44491a94b32067ba681ad: Bootstrap starting.
I20260808 22:21:58.151321 13386 tablet_bootstrap.cc:654] T a9ad20cd392f4b168b74a92d991c4b9a P ec50da59889f4a1e84a835ff268b75e8: Neither blocks nor log segments found. Creating new log.
I20260808 22:21:58.151441 13389 tablet_bootstrap.cc:654] T a9ad20cd392f4b168b74a92d991c4b9a P b9c4150963e04e018ba397151c05dc61: Neither blocks nor log segments found. Creating new log.
I20260808 22:21:58.151906 13390 tablet_bootstrap.cc:654] T a9ad20cd392f4b168b74a92d991c4b9a P bcba8d834ed44491a94b32067ba681ad: Neither blocks nor log segments found. Creating new log.
I20260808 22:21:58.152484 13389 tablet_bootstrap.cc:492] T a9ad20cd392f4b168b74a92d991c4b9a P b9c4150963e04e018ba397151c05dc61: No bootstrap required, opened a new log
I20260808 22:21:58.152518 13386 tablet_bootstrap.cc:492] T a9ad20cd392f4b168b74a92d991c4b9a P ec50da59889f4a1e84a835ff268b75e8: No bootstrap required, opened a new log
I20260808 22:21:58.152577 13389 ts_tablet_manager.cc:1397] T a9ad20cd392f4b168b74a92d991c4b9a P b9c4150963e04e018ba397151c05dc61: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 22:21:58.152597 13386 ts_tablet_manager.cc:1397] T a9ad20cd392f4b168b74a92d991c4b9a P ec50da59889f4a1e84a835ff268b75e8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 22:21:58.153044 13386 raft_consensus.cc:348] T a9ad20cd392f4b168b74a92d991c4b9a P ec50da59889f4a1e84a835ff268b75e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec50da59889f4a1e84a835ff268b75e8" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34469 } } peers { permanent_uuid: "bcba8d834ed44491a94b32067ba681ad" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36839 } } peers { permanent_uuid: "b9c4150963e04e018ba397151c05dc61" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45545 } }
I20260808 22:21:58.153157 13386 raft_consensus.cc:374] T a9ad20cd392f4b168b74a92d991c4b9a P ec50da59889f4a1e84a835ff268b75e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:21:58.153195 13386 raft_consensus.cc:729] T a9ad20cd392f4b168b74a92d991c4b9a P ec50da59889f4a1e84a835ff268b75e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec50da59889f4a1e84a835ff268b75e8, State: Initialized, Role: FOLLOWER
I20260808 22:21:58.153230 13389 raft_consensus.cc:348] T a9ad20cd392f4b168b74a92d991c4b9a P b9c4150963e04e018ba397151c05dc61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec50da59889f4a1e84a835ff268b75e8" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34469 } } peers { permanent_uuid: "bcba8d834ed44491a94b32067ba681ad" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36839 } } peers { permanent_uuid: "b9c4150963e04e018ba397151c05dc61" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45545 } }
I20260808 22:21:58.153345 13389 raft_consensus.cc:374] T a9ad20cd392f4b168b74a92d991c4b9a P b9c4150963e04e018ba397151c05dc61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:21:58.153378 13389 raft_consensus.cc:729] T a9ad20cd392f4b168b74a92d991c4b9a P b9c4150963e04e018ba397151c05dc61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9c4150963e04e018ba397151c05dc61, State: Initialized, Role: FOLLOWER
I20260808 22:21:58.153326 13386 consensus_queue.cc:260] T a9ad20cd392f4b168b74a92d991c4b9a P ec50da59889f4a1e84a835ff268b75e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec50da59889f4a1e84a835ff268b75e8" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34469 } } peers { permanent_uuid: "bcba8d834ed44491a94b32067ba681ad" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36839 } } peers { permanent_uuid: "b9c4150963e04e018ba397151c05dc61" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45545 } }
I20260808 22:21:58.153527 13389 consensus_queue.cc:260] T a9ad20cd392f4b168b74a92d991c4b9a P b9c4150963e04e018ba397151c05dc61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec50da59889f4a1e84a835ff268b75e8" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34469 } } peers { permanent_uuid: "bcba8d834ed44491a94b32067ba681ad" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36839 } } peers { permanent_uuid: "b9c4150963e04e018ba397151c05dc61" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45545 } }
I20260808 22:21:58.153595 13386 ts_tablet_manager.cc:1428] T a9ad20cd392f4b168b74a92d991c4b9a P ec50da59889f4a1e84a835ff268b75e8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 22:21:58.153890 13389 ts_tablet_manager.cc:1428] T a9ad20cd392f4b168b74a92d991c4b9a P b9c4150963e04e018ba397151c05dc61: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 22:21:58.154385 13390 tablet_bootstrap.cc:492] T a9ad20cd392f4b168b74a92d991c4b9a P bcba8d834ed44491a94b32067ba681ad: No bootstrap required, opened a new log
I20260808 22:21:58.154477 13390 ts_tablet_manager.cc:1397] T a9ad20cd392f4b168b74a92d991c4b9a P bcba8d834ed44491a94b32067ba681ad: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 22:21:58.154844 13390 raft_consensus.cc:348] T a9ad20cd392f4b168b74a92d991c4b9a P bcba8d834ed44491a94b32067ba681ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec50da59889f4a1e84a835ff268b75e8" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34469 } } peers { permanent_uuid: "bcba8d834ed44491a94b32067ba681ad" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36839 } } peers { permanent_uuid: "b9c4150963e04e018ba397151c05dc61" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45545 } }
I20260808 22:21:58.154932 13390 raft_consensus.cc:374] T a9ad20cd392f4b168b74a92d991c4b9a P bcba8d834ed44491a94b32067ba681ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:21:58.154960 13390 raft_consensus.cc:729] T a9ad20cd392f4b168b74a92d991c4b9a P bcba8d834ed44491a94b32067ba681ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcba8d834ed44491a94b32067ba681ad, State: Initialized, Role: FOLLOWER
I20260808 22:21:58.155077 13390 consensus_queue.cc:260] T a9ad20cd392f4b168b74a92d991c4b9a P bcba8d834ed44491a94b32067ba681ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec50da59889f4a1e84a835ff268b75e8" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34469 } } peers { permanent_uuid: "bcba8d834ed44491a94b32067ba681ad" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36839 } } peers { permanent_uuid: "b9c4150963e04e018ba397151c05dc61" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45545 } }
I20260808 22:21:58.155468 13390 ts_tablet_manager.cc:1428] T a9ad20cd392f4b168b74a92d991c4b9a P bcba8d834ed44491a94b32067ba681ad: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 22:21:58.298854 13361 raft_consensus.cc:482] T a9ad20cd392f4b168b74a92d991c4b9a P ec50da59889f4a1e84a835ff268b75e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 22:21:58.299027 13361 raft_consensus.cc:504] T a9ad20cd392f4b168b74a92d991c4b9a P ec50da59889f4a1e84a835ff268b75e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec50da59889f4a1e84a835ff268b75e8" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34469 } } peers { permanent_uuid: "bcba8d834ed44491a94b32067ba681ad" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36839 } } peers { permanent_uuid: "b9c4150963e04e018ba397151c05dc61" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45545 } }
I20260808 22:21:58.299516 13361 leader_election.cc:288] T a9ad20cd392f4b168b74a92d991c4b9a P ec50da59889f4a1e84a835ff268b75e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bcba8d834ed44491a94b32067ba681ad (127.9.176.194:36839), b9c4150963e04e018ba397151c05dc61 (127.9.176.195:45545)
I20260808 22:21:58.300035 12928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9ad20cd392f4b168b74a92d991c4b9a" candidate_uuid: "ec50da59889f4a1e84a835ff268b75e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcba8d834ed44491a94b32067ba681ad" is_pre_election: true
I20260808 22:21:58.300046 13144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9ad20cd392f4b168b74a92d991c4b9a" candidate_uuid: "ec50da59889f4a1e84a835ff268b75e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9c4150963e04e018ba397151c05dc61" is_pre_election: true
I20260808 22:21:58.300202 13144 raft_consensus.cc:2454] T a9ad20cd392f4b168b74a92d991c4b9a P b9c4150963e04e018ba397151c05dc61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec50da59889f4a1e84a835ff268b75e8 in term 0.
I20260808 22:21:58.300202 12928 raft_consensus.cc:2454] T a9ad20cd392f4b168b74a92d991c4b9a P bcba8d834ed44491a94b32067ba681ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec50da59889f4a1e84a835ff268b75e8 in term 0.
I20260808 22:21:58.300560 12610 leader_election.cc:302] T a9ad20cd392f4b168b74a92d991c4b9a P ec50da59889f4a1e84a835ff268b75e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: b9c4150963e04e018ba397151c05dc61, bcba8d834ed44491a94b32067ba681ad, ec50da59889f4a1e84a835ff268b75e8; no voters:
I20260808 22:21:58.300766 13361 raft_consensus.cc:2781] T a9ad20cd392f4b168b74a92d991c4b9a P ec50da59889f4a1e84a835ff268b75e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 22:21:58.300839 13361 raft_consensus.cc:482] T a9ad20cd392f4b168b74a92d991c4b9a P ec50da59889f4a1e84a835ff268b75e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 22:21:58.300877 13361 raft_consensus.cc:3037] T a9ad20cd392f4b168b74a92d991c4b9a P ec50da59889f4a1e84a835ff268b75e8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:21:58.302006 13361 raft_consensus.cc:504] T a9ad20cd392f4b168b74a92d991c4b9a P ec50da59889f4a1e84a835ff268b75e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec50da59889f4a1e84a835ff268b75e8" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34469 } } peers { permanent_uuid: "bcba8d834ed44491a94b32067ba681ad" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36839 } } peers { permanent_uuid: "b9c4150963e04e018ba397151c05dc61" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45545 } }
I20260808 22:21:58.302328 13361 leader_election.cc:288] T a9ad20cd392f4b168b74a92d991c4b9a P ec50da59889f4a1e84a835ff268b75e8 [CANDIDATE]: Term 1 election: Requested vote from peers bcba8d834ed44491a94b32067ba681ad (127.9.176.194:36839), b9c4150963e04e018ba397151c05dc61 (127.9.176.195:45545)
I20260808 22:21:58.302657 13144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9ad20cd392f4b168b74a92d991c4b9a" candidate_uuid: "ec50da59889f4a1e84a835ff268b75e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9c4150963e04e018ba397151c05dc61"
I20260808 22:21:58.302680 12928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9ad20cd392f4b168b74a92d991c4b9a" candidate_uuid: "ec50da59889f4a1e84a835ff268b75e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcba8d834ed44491a94b32067ba681ad"
I20260808 22:21:58.302768 13144 raft_consensus.cc:3037] T a9ad20cd392f4b168b74a92d991c4b9a P b9c4150963e04e018ba397151c05dc61 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:21:58.302819 12928 raft_consensus.cc:3037] T a9ad20cd392f4b168b74a92d991c4b9a P bcba8d834ed44491a94b32067ba681ad [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:21:58.303673 13144 raft_consensus.cc:2454] T a9ad20cd392f4b168b74a92d991c4b9a P b9c4150963e04e018ba397151c05dc61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec50da59889f4a1e84a835ff268b75e8 in term 1.
I20260808 22:21:58.303819 12928 raft_consensus.cc:2454] T a9ad20cd392f4b168b74a92d991c4b9a P bcba8d834ed44491a94b32067ba681ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec50da59889f4a1e84a835ff268b75e8 in term 1.
I20260808 22:21:58.303929 12610 leader_election.cc:302] T a9ad20cd392f4b168b74a92d991c4b9a P ec50da59889f4a1e84a835ff268b75e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9c4150963e04e018ba397151c05dc61, ec50da59889f4a1e84a835ff268b75e8; no voters:
I20260808 22:21:58.304147 13361 raft_consensus.cc:2781] T a9ad20cd392f4b168b74a92d991c4b9a P ec50da59889f4a1e84a835ff268b75e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:21:58.304241 13361 raft_consensus.cc:686] T a9ad20cd392f4b168b74a92d991c4b9a P ec50da59889f4a1e84a835ff268b75e8 [term 1 LEADER]: Becoming Leader. State: Replica: ec50da59889f4a1e84a835ff268b75e8, State: Running, Role: LEADER
I20260808 22:21:58.304394 13361 consensus_queue.cc:237] T a9ad20cd392f4b168b74a92d991c4b9a P ec50da59889f4a1e84a835ff268b75e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec50da59889f4a1e84a835ff268b75e8" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34469 } } peers { permanent_uuid: "bcba8d834ed44491a94b32067ba681ad" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36839 } } peers { permanent_uuid: "b9c4150963e04e018ba397151c05dc61" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45545 } }
I20260808 22:21:58.306147 12465 catalog_manager.cc:5495] T a9ad20cd392f4b168b74a92d991c4b9a P ec50da59889f4a1e84a835ff268b75e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec50da59889f4a1e84a835ff268b75e8 (127.9.176.193). New cstate: current_term: 1 leader_uuid: "ec50da59889f4a1e84a835ff268b75e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec50da59889f4a1e84a835ff268b75e8" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcba8d834ed44491a94b32067ba681ad" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36839 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9c4150963e04e018ba397151c05dc61" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45545 } health_report { overall_health: UNKNOWN } } }
I20260808 22:21:58.349339 13409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:58.346448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45468 (local address 127.9.176.193:34469)
0808 22:21:58.346700 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0808 22:21:58.346708 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 22:21:58.346721 (+ 13us) server_negotiation.cc:407] Connection header received
0808 22:21:58.346762 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:21:58.346764 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:21:58.346808 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:21:58.346886 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:21:58.347085 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:58.347631 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:58.348447 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:58.348700 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:58.348784 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:58.348999 (+ 215us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:21:58.349063 (+ 64us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:21:58.349109 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0808 22:21:58.349176 (+ 67us) server_negotiation.cc:299] Negotiation successful
0808 22:21:58.349218 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":105,"threads_started":1}
I20260808 22:21:58.354480 13144 raft_consensus.cc:1261] T a9ad20cd392f4b168b74a92d991c4b9a P b9c4150963e04e018ba397151c05dc61 [term 1 FOLLOWER]: Refusing update from remote peer ec50da59889f4a1e84a835ff268b75e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:21:58.354480 12928 raft_consensus.cc:1261] T a9ad20cd392f4b168b74a92d991c4b9a P bcba8d834ed44491a94b32067ba681ad [term 1 FOLLOWER]: Refusing update from remote peer ec50da59889f4a1e84a835ff268b75e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:21:58.354846 13361 consensus_queue.cc:1035] T a9ad20cd392f4b168b74a92d991c4b9a P ec50da59889f4a1e84a835ff268b75e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcba8d834ed44491a94b32067ba681ad" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:21:58.355026 13340 consensus_queue.cc:1035] T a9ad20cd392f4b168b74a92d991c4b9a P ec50da59889f4a1e84a835ff268b75e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9c4150963e04e018ba397151c05dc61" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:21:58.382094 12465 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40638:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:21:58.382246 12465 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40638:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:21:58.384040 12465 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a9a81a719cd649ec800f8a3cc702574d: Sending DeleteTablet for 3 replicas of tablet a9ad20cd392f4b168b74a92d991c4b9a
I20260808 22:21:58.384502 12681 tablet_service.cc:1514] Processing DeleteTablet for tablet a9ad20cd392f4b168b74a92d991c4b9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:21:58 UTC) from {username='kudu', principal='kudu/127.9.176.254@KRBTEST.COM'} at 127.0.0.1:45434
I20260808 22:21:58.384604 12904 tablet_service.cc:1514] Processing DeleteTablet for tablet a9ad20cd392f4b168b74a92d991c4b9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:21:58 UTC) from {username='kudu', principal='kudu/127.9.176.254@KRBTEST.COM'} at 127.0.0.1:46780
I20260808 22:21:58.384603 13113 tablet_service.cc:1514] Processing DeleteTablet for tablet a9ad20cd392f4b168b74a92d991c4b9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:21:58 UTC) from {username='kudu', principal='kudu/127.9.176.254@KRBTEST.COM'} at 127.0.0.1:50896
I20260808 22:21:58.384692 13370 tablet_replica.cc:330] T a9ad20cd392f4b168b74a92d991c4b9a P ec50da59889f4a1e84a835ff268b75e8: stopping tablet replica
I20260808 22:21:58.384790 13371 tablet_replica.cc:330] T a9ad20cd392f4b168b74a92d991c4b9a P bcba8d834ed44491a94b32067ba681ad: stopping tablet replica
I20260808 22:21:58.384800 13370 raft_consensus.cc:2229] T a9ad20cd392f4b168b74a92d991c4b9a P ec50da59889f4a1e84a835ff268b75e8 [term 1 LEADER]: Raft consensus shutting down.
I20260808 22:21:58.384912 13371 raft_consensus.cc:2229] T a9ad20cd392f4b168b74a92d991c4b9a P bcba8d834ed44491a94b32067ba681ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:21:58.384991 13370 raft_consensus.cc:2258] T a9ad20cd392f4b168b74a92d991c4b9a P ec50da59889f4a1e84a835ff268b75e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:21:58.385011 13371 raft_consensus.cc:2258] T a9ad20cd392f4b168b74a92d991c4b9a P bcba8d834ed44491a94b32067ba681ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:21:58.385051 13373 tablet_replica.cc:330] T a9ad20cd392f4b168b74a92d991c4b9a P b9c4150963e04e018ba397151c05dc61: stopping tablet replica
I20260808 22:21:58.385161 13373 raft_consensus.cc:2229] T a9ad20cd392f4b168b74a92d991c4b9a P b9c4150963e04e018ba397151c05dc61 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:21:58.385281 13373 raft_consensus.cc:2258] T a9ad20cd392f4b168b74a92d991c4b9a P b9c4150963e04e018ba397151c05dc61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:21:58.385756 13370 ts_tablet_manager.cc:1905] T a9ad20cd392f4b168b74a92d991c4b9a P ec50da59889f4a1e84a835ff268b75e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:21:58.385955 12465 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40638:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260808 22:21:58.386078 13373 ts_tablet_manager.cc:1905] T a9ad20cd392f4b168b74a92d991c4b9a P b9c4150963e04e018ba397151c05dc61: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260808 22:21:58.386329 12465 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 22:21:58.388023 13370 ts_tablet_manager.cc:1918] T a9ad20cd392f4b168b74a92d991c4b9a P ec50da59889f4a1e84a835ff268b75e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 22:21:58.388075 13370 log.cc:1196] T a9ad20cd392f4b168b74a92d991c4b9a P ec50da59889f4a1e84a835ff268b75e8: Deleting WAL directory at /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/minicluster-data/ts-0/wal/wals/a9ad20cd392f4b168b74a92d991c4b9a
I20260808 22:21:58.388244 13370 ts_tablet_manager.cc:1939] T a9ad20cd392f4b168b74a92d991c4b9a P ec50da59889f4a1e84a835ff268b75e8: Deleting consensus metadata
I20260808 22:21:58.388578 13373 ts_tablet_manager.cc:1918] T a9ad20cd392f4b168b74a92d991c4b9a P b9c4150963e04e018ba397151c05dc61: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 22:21:58.388648 13373 log.cc:1196] T a9ad20cd392f4b168b74a92d991c4b9a P b9c4150963e04e018ba397151c05dc61: Deleting WAL directory at /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/minicluster-data/ts-2/wal/wals/a9ad20cd392f4b168b74a92d991c4b9a
I20260808 22:21:58.388793 13371 ts_tablet_manager.cc:1905] T a9ad20cd392f4b168b74a92d991c4b9a P bcba8d834ed44491a94b32067ba681ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:21:58.388876 13373 ts_tablet_manager.cc:1939] T a9ad20cd392f4b168b74a92d991c4b9a P b9c4150963e04e018ba397151c05dc61: Deleting consensus metadata
I20260808 22:21:58.389578 12449 catalog_manager.cc:4841] TS b9c4150963e04e018ba397151c05dc61 (127.9.176.195:45545): tablet a9ad20cd392f4b168b74a92d991c4b9a (table test-table [id=790328f1386d455c8fa36b7bc0318e4b]) successfully deleted
I20260808 22:21:58.390158 12450 catalog_manager.cc:4841] TS ec50da59889f4a1e84a835ff268b75e8 (127.9.176.193:34469): tablet a9ad20cd392f4b168b74a92d991c4b9a (table test-table [id=790328f1386d455c8fa36b7bc0318e4b]) successfully deleted
I20260808 22:21:58.390918 12904 tablet_service.cc:1467] Processing CreateTablet for tablet c7dfae141a664ba780dd922f3e932998 (DEFAULT_TABLE table=test-table [id=b5521f79c46d43249ca074f6772d6da6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:21:58.390908 12681 tablet_service.cc:1467] Processing CreateTablet for tablet c7dfae141a664ba780dd922f3e932998 (DEFAULT_TABLE table=test-table [id=b5521f79c46d43249ca074f6772d6da6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:21:58.391090 13371 ts_tablet_manager.cc:1918] T a9ad20cd392f4b168b74a92d991c4b9a P bcba8d834ed44491a94b32067ba681ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 22:21:58.391149 12681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7dfae141a664ba780dd922f3e932998. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:21:58.391160 12904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7dfae141a664ba780dd922f3e932998. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:21:58.392933 13386 tablet_bootstrap.cc:492] T c7dfae141a664ba780dd922f3e932998 P ec50da59889f4a1e84a835ff268b75e8: Bootstrap starting.
I20260808 22:21:58.392969 13390 tablet_bootstrap.cc:492] T c7dfae141a664ba780dd922f3e932998 P bcba8d834ed44491a94b32067ba681ad: Bootstrap starting.
I20260808 22:21:58.391167 13371 log.cc:1196] T a9ad20cd392f4b168b74a92d991c4b9a P bcba8d834ed44491a94b32067ba681ad: Deleting WAL directory at /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/minicluster-data/ts-1/wal/wals/a9ad20cd392f4b168b74a92d991c4b9a
I20260808 22:21:58.393110 13113 tablet_service.cc:1467] Processing CreateTablet for tablet c7dfae141a664ba780dd922f3e932998 (DEFAULT_TABLE table=test-table [id=b5521f79c46d43249ca074f6772d6da6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:21:58.393285 13371 ts_tablet_manager.cc:1939] T a9ad20cd392f4b168b74a92d991c4b9a P bcba8d834ed44491a94b32067ba681ad: Deleting consensus metadata
I20260808 22:21:58.393329 13113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7dfae141a664ba780dd922f3e932998. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:21:58.393708 13386 tablet_bootstrap.cc:654] T c7dfae141a664ba780dd922f3e932998 P ec50da59889f4a1e84a835ff268b75e8: Neither blocks nor log segments found. Creating new log.
I20260808 22:21:58.393868 13390 tablet_bootstrap.cc:654] T c7dfae141a664ba780dd922f3e932998 P bcba8d834ed44491a94b32067ba681ad: Neither blocks nor log segments found. Creating new log.
I20260808 22:21:58.394981 13390 tablet_bootstrap.cc:492] T c7dfae141a664ba780dd922f3e932998 P bcba8d834ed44491a94b32067ba681ad: No bootstrap required, opened a new log
I20260808 22:21:58.395074 13390 ts_tablet_manager.cc:1397] T c7dfae141a664ba780dd922f3e932998 P bcba8d834ed44491a94b32067ba681ad: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 22:21:58.395303 13389 tablet_bootstrap.cc:492] T c7dfae141a664ba780dd922f3e932998 P b9c4150963e04e018ba397151c05dc61: Bootstrap starting.
I20260808 22:21:58.395421 12448 catalog_manager.cc:4841] TS bcba8d834ed44491a94b32067ba681ad (127.9.176.194:36839): tablet a9ad20cd392f4b168b74a92d991c4b9a (table test-table [id=790328f1386d455c8fa36b7bc0318e4b]) successfully deleted
I20260808 22:21:58.395481 13390 raft_consensus.cc:348] T c7dfae141a664ba780dd922f3e932998 P bcba8d834ed44491a94b32067ba681ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec50da59889f4a1e84a835ff268b75e8" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34469 } } peers { permanent_uuid: "bcba8d834ed44491a94b32067ba681ad" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36839 } } peers { permanent_uuid: "b9c4150963e04e018ba397151c05dc61" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45545 } }
I20260808 22:21:58.395578 13390 raft_consensus.cc:374] T c7dfae141a664ba780dd922f3e932998 P bcba8d834ed44491a94b32067ba681ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:21:58.395608 13390 raft_consensus.cc:729] T c7dfae141a664ba780dd922f3e932998 P bcba8d834ed44491a94b32067ba681ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcba8d834ed44491a94b32067ba681ad, State: Initialized, Role: FOLLOWER
I20260808 22:21:58.395722 13390 consensus_queue.cc:260] T c7dfae141a664ba780dd922f3e932998 P bcba8d834ed44491a94b32067ba681ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec50da59889f4a1e84a835ff268b75e8" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34469 } } peers { permanent_uuid: "bcba8d834ed44491a94b32067ba681ad" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36839 } } peers { permanent_uuid: "b9c4150963e04e018ba397151c05dc61" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45545 } }
I20260808 22:21:58.395962 13390 ts_tablet_manager.cc:1428] T c7dfae141a664ba780dd922f3e932998 P bcba8d834ed44491a94b32067ba681ad: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 22:21:58.396111 13389 tablet_bootstrap.cc:654] T c7dfae141a664ba780dd922f3e932998 P b9c4150963e04e018ba397151c05dc61: Neither blocks nor log segments found. Creating new log.
I20260808 22:21:58.396178 13386 tablet_bootstrap.cc:492] T c7dfae141a664ba780dd922f3e932998 P ec50da59889f4a1e84a835ff268b75e8: No bootstrap required, opened a new log
I20260808 22:21:58.396250 13386 ts_tablet_manager.cc:1397] T c7dfae141a664ba780dd922f3e932998 P ec50da59889f4a1e84a835ff268b75e8: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260808 22:21:58.396632 13386 raft_consensus.cc:348] T c7dfae141a664ba780dd922f3e932998 P ec50da59889f4a1e84a835ff268b75e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec50da59889f4a1e84a835ff268b75e8" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34469 } } peers { permanent_uuid: "bcba8d834ed44491a94b32067ba681ad" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36839 } } peers { permanent_uuid: "b9c4150963e04e018ba397151c05dc61" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45545 } }
I20260808 22:21:58.396731 13386 raft_consensus.cc:374] T c7dfae141a664ba780dd922f3e932998 P ec50da59889f4a1e84a835ff268b75e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:21:58.396761 13386 raft_consensus.cc:729] T c7dfae141a664ba780dd922f3e932998 P ec50da59889f4a1e84a835ff268b75e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec50da59889f4a1e84a835ff268b75e8, State: Initialized, Role: FOLLOWER
I20260808 22:21:58.396874 13386 consensus_queue.cc:260] T c7dfae141a664ba780dd922f3e932998 P ec50da59889f4a1e84a835ff268b75e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec50da59889f4a1e84a835ff268b75e8" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34469 } } peers { permanent_uuid: "bcba8d834ed44491a94b32067ba681ad" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36839 } } peers { permanent_uuid: "b9c4150963e04e018ba397151c05dc61" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45545 } }
I20260808 22:21:58.397156 13386 ts_tablet_manager.cc:1428] T c7dfae141a664ba780dd922f3e932998 P ec50da59889f4a1e84a835ff268b75e8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 22:21:58.397277 13389 tablet_bootstrap.cc:492] T c7dfae141a664ba780dd922f3e932998 P b9c4150963e04e018ba397151c05dc61: No bootstrap required, opened a new log
I20260808 22:21:58.397363 13389 ts_tablet_manager.cc:1397] T c7dfae141a664ba780dd922f3e932998 P b9c4150963e04e018ba397151c05dc61: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 22:21:58.397749 13389 raft_consensus.cc:348] T c7dfae141a664ba780dd922f3e932998 P b9c4150963e04e018ba397151c05dc61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec50da59889f4a1e84a835ff268b75e8" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34469 } } peers { permanent_uuid: "bcba8d834ed44491a94b32067ba681ad" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36839 } } peers { permanent_uuid: "b9c4150963e04e018ba397151c05dc61" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45545 } }
I20260808 22:21:58.397859 13389 raft_consensus.cc:374] T c7dfae141a664ba780dd922f3e932998 P b9c4150963e04e018ba397151c05dc61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:21:58.397892 13389 raft_consensus.cc:729] T c7dfae141a664ba780dd922f3e932998 P b9c4150963e04e018ba397151c05dc61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9c4150963e04e018ba397151c05dc61, State: Initialized, Role: FOLLOWER
I20260808 22:21:58.398008 13389 consensus_queue.cc:260] T c7dfae141a664ba780dd922f3e932998 P b9c4150963e04e018ba397151c05dc61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec50da59889f4a1e84a835ff268b75e8" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34469 } } peers { permanent_uuid: "bcba8d834ed44491a94b32067ba681ad" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36839 } } peers { permanent_uuid: "b9c4150963e04e018ba397151c05dc61" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45545 } }
I20260808 22:21:58.398224 13389 ts_tablet_manager.cc:1428] T c7dfae141a664ba780dd922f3e932998 P b9c4150963e04e018ba397151c05dc61: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260808 22:21:58.497576 13397 raft_consensus.cc:482] T c7dfae141a664ba780dd922f3e932998 P b9c4150963e04e018ba397151c05dc61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 22:21:58.497733 13397 raft_consensus.cc:504] T c7dfae141a664ba780dd922f3e932998 P b9c4150963e04e018ba397151c05dc61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec50da59889f4a1e84a835ff268b75e8" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34469 } } peers { permanent_uuid: "bcba8d834ed44491a94b32067ba681ad" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36839 } } peers { permanent_uuid: "b9c4150963e04e018ba397151c05dc61" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45545 } }
I20260808 22:21:58.498155 13397 leader_election.cc:288] T c7dfae141a664ba780dd922f3e932998 P b9c4150963e04e018ba397151c05dc61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec50da59889f4a1e84a835ff268b75e8 (127.9.176.193:34469), bcba8d834ed44491a94b32067ba681ad (127.9.176.194:36839)
I20260808 22:21:58.498553 12716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7dfae141a664ba780dd922f3e932998" candidate_uuid: "b9c4150963e04e018ba397151c05dc61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec50da59889f4a1e84a835ff268b75e8" is_pre_election: true
I20260808 22:21:58.498647 12928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7dfae141a664ba780dd922f3e932998" candidate_uuid: "b9c4150963e04e018ba397151c05dc61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcba8d834ed44491a94b32067ba681ad" is_pre_election: true
I20260808 22:21:58.498718 12716 raft_consensus.cc:2454] T c7dfae141a664ba780dd922f3e932998 P ec50da59889f4a1e84a835ff268b75e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9c4150963e04e018ba397151c05dc61 in term 0.
I20260808 22:21:58.498785 12928 raft_consensus.cc:2454] T c7dfae141a664ba780dd922f3e932998 P bcba8d834ed44491a94b32067ba681ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9c4150963e04e018ba397151c05dc61 in term 0.
I20260808 22:21:58.499058 13045 leader_election.cc:302] T c7dfae141a664ba780dd922f3e932998 P b9c4150963e04e018ba397151c05dc61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: b9c4150963e04e018ba397151c05dc61, bcba8d834ed44491a94b32067ba681ad, ec50da59889f4a1e84a835ff268b75e8; no voters:
I20260808 22:21:58.499254 13397 raft_consensus.cc:2781] T c7dfae141a664ba780dd922f3e932998 P b9c4150963e04e018ba397151c05dc61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 22:21:58.499332 13397 raft_consensus.cc:482] T c7dfae141a664ba780dd922f3e932998 P b9c4150963e04e018ba397151c05dc61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 22:21:58.499374 13397 raft_consensus.cc:3037] T c7dfae141a664ba780dd922f3e932998 P b9c4150963e04e018ba397151c05dc61 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:21:58.500439 13397 raft_consensus.cc:504] T c7dfae141a664ba780dd922f3e932998 P b9c4150963e04e018ba397151c05dc61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec50da59889f4a1e84a835ff268b75e8" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34469 } } peers { permanent_uuid: "bcba8d834ed44491a94b32067ba681ad" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36839 } } peers { permanent_uuid: "b9c4150963e04e018ba397151c05dc61" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45545 } }
I20260808 22:21:58.500749 13397 leader_election.cc:288] T c7dfae141a664ba780dd922f3e932998 P b9c4150963e04e018ba397151c05dc61 [CANDIDATE]: Term 1 election: Requested vote from peers ec50da59889f4a1e84a835ff268b75e8 (127.9.176.193:34469), bcba8d834ed44491a94b32067ba681ad (127.9.176.194:36839)
I20260808 22:21:58.501108 12928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7dfae141a664ba780dd922f3e932998" candidate_uuid: "b9c4150963e04e018ba397151c05dc61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcba8d834ed44491a94b32067ba681ad"
I20260808 22:21:58.501101 12716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7dfae141a664ba780dd922f3e932998" candidate_uuid: "b9c4150963e04e018ba397151c05dc61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec50da59889f4a1e84a835ff268b75e8"
I20260808 22:21:58.501215 12716 raft_consensus.cc:3037] T c7dfae141a664ba780dd922f3e932998 P ec50da59889f4a1e84a835ff268b75e8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:21:58.501224 12928 raft_consensus.cc:3037] T c7dfae141a664ba780dd922f3e932998 P bcba8d834ed44491a94b32067ba681ad [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:21:58.502069 12928 raft_consensus.cc:2454] T c7dfae141a664ba780dd922f3e932998 P bcba8d834ed44491a94b32067ba681ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9c4150963e04e018ba397151c05dc61 in term 1.
I20260808 22:21:58.502133 12716 raft_consensus.cc:2454] T c7dfae141a664ba780dd922f3e932998 P ec50da59889f4a1e84a835ff268b75e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9c4150963e04e018ba397151c05dc61 in term 1.
I20260808 22:21:58.502334 13035 leader_election.cc:302] T c7dfae141a664ba780dd922f3e932998 P b9c4150963e04e018ba397151c05dc61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9c4150963e04e018ba397151c05dc61, bcba8d834ed44491a94b32067ba681ad; no voters:
I20260808 22:21:58.502537 13397 raft_consensus.cc:2781] T c7dfae141a664ba780dd922f3e932998 P b9c4150963e04e018ba397151c05dc61 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:21:58.502689 13397 raft_consensus.cc:686] T c7dfae141a664ba780dd922f3e932998 P b9c4150963e04e018ba397151c05dc61 [term 1 LEADER]: Becoming Leader. State: Replica: b9c4150963e04e018ba397151c05dc61, State: Running, Role: LEADER
I20260808 22:21:58.502832 13397 consensus_queue.cc:237] T c7dfae141a664ba780dd922f3e932998 P b9c4150963e04e018ba397151c05dc61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec50da59889f4a1e84a835ff268b75e8" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34469 } } peers { permanent_uuid: "bcba8d834ed44491a94b32067ba681ad" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36839 } } peers { permanent_uuid: "b9c4150963e04e018ba397151c05dc61" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45545 } }
I20260808 22:21:58.504396 12465 catalog_manager.cc:5495] T c7dfae141a664ba780dd922f3e932998 P b9c4150963e04e018ba397151c05dc61 reported cstate change: term changed from 0 to 1, leader changed from <none> to b9c4150963e04e018ba397151c05dc61 (127.9.176.195). New cstate: current_term: 1 leader_uuid: "b9c4150963e04e018ba397151c05dc61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec50da59889f4a1e84a835ff268b75e8" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcba8d834ed44491a94b32067ba681ad" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36839 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9c4150963e04e018ba397151c05dc61" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45545 } health_report { overall_health: HEALTHY } } }
I20260808 22:21:58.526605 13420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:21:58.523758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50918 (local address 127.9.176.195:45545)
0808 22:21:58.524047 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0808 22:21:58.524070 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0808 22:21:58.524081 (+ 11us) server_negotiation.cc:407] Connection header received
0808 22:21:58.524118 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:21:58.524120 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:21:58.524158 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:21:58.524222 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:21:58.524445 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:58.525060 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:58.525799 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:21:58.526023 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:21:58.526092 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:21:58.526255 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:21:58.526316 (+ 61us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:21:58.526362 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0808 22:21:58.526419 (+ 57us) server_negotiation.cc:299] Negotiation successful
0808 22:21:58.526466 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":120,"threads_started":1}
I20260808 22:21:58.531749 12716 raft_consensus.cc:1261] T c7dfae141a664ba780dd922f3e932998 P ec50da59889f4a1e84a835ff268b75e8 [term 1 FOLLOWER]: Refusing update from remote peer b9c4150963e04e018ba397151c05dc61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:21:58.531749 12928 raft_consensus.cc:1261] T c7dfae141a664ba780dd922f3e932998 P bcba8d834ed44491a94b32067ba681ad [term 1 FOLLOWER]: Refusing update from remote peer b9c4150963e04e018ba397151c05dc61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:21:58.532122 13418 consensus_queue.cc:1035] T c7dfae141a664ba780dd922f3e932998 P b9c4150963e04e018ba397151c05dc61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec50da59889f4a1e84a835ff268b75e8" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:21:58.532260 13397 consensus_queue.cc:1035] T c7dfae141a664ba780dd922f3e932998 P b9c4150963e04e018ba397151c05dc61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcba8d834ed44491a94b32067ba681ad" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 22:21:58.544486 13341 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260808 22:22:00.100461 12465 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40638:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:22:00.100651 12465 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40638:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:22:00.102965 12465 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a9a81a719cd649ec800f8a3cc702574d: Sending DeleteTablet for 3 replicas of tablet c7dfae141a664ba780dd922f3e932998
I20260808 22:22:00.103466 12681 tablet_service.cc:1514] Processing DeleteTablet for tablet c7dfae141a664ba780dd922f3e932998 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:22:00 UTC) from {username='kudu', principal='kudu/127.9.176.254@KRBTEST.COM'} at 127.0.0.1:45434
I20260808 22:22:00.103518 12904 tablet_service.cc:1514] Processing DeleteTablet for tablet c7dfae141a664ba780dd922f3e932998 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:22:00 UTC) from {username='kudu', principal='kudu/127.9.176.254@KRBTEST.COM'} at 127.0.0.1:46780
I20260808 22:22:00.103564 13113 tablet_service.cc:1514] Processing DeleteTablet for tablet c7dfae141a664ba780dd922f3e932998 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:22:00 UTC) from {username='kudu', principal='kudu/127.9.176.254@KRBTEST.COM'} at 127.0.0.1:50896
I20260808 22:22:00.103839 13462 tablet_replica.cc:330] T c7dfae141a664ba780dd922f3e932998 P bcba8d834ed44491a94b32067ba681ad: stopping tablet replica
I20260808 22:22:00.103842 13461 tablet_replica.cc:330] T c7dfae141a664ba780dd922f3e932998 P ec50da59889f4a1e84a835ff268b75e8: stopping tablet replica
I20260808 22:22:00.103911 13463 tablet_replica.cc:330] T c7dfae141a664ba780dd922f3e932998 P b9c4150963e04e018ba397151c05dc61: stopping tablet replica
I20260808 22:22:00.103967 13461 raft_consensus.cc:2229] T c7dfae141a664ba780dd922f3e932998 P ec50da59889f4a1e84a835ff268b75e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:22:00.103984 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 12585
I20260808 22:22:00.108924 13462 raft_consensus.cc:2229] T c7dfae141a664ba780dd922f3e932998 P bcba8d834ed44491a94b32067ba681ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:22:00.109174 13462 raft_consensus.cc:2258] T c7dfae141a664ba780dd922f3e932998 P bcba8d834ed44491a94b32067ba681ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:22:00.104072 13463 raft_consensus.cc:2229] T c7dfae141a664ba780dd922f3e932998 P b9c4150963e04e018ba397151c05dc61 [term 1 LEADER]: Raft consensus shutting down.
I20260808 22:22:00.110719 13463 raft_consensus.cc:2258] T c7dfae141a664ba780dd922f3e932998 P b9c4150963e04e018ba397151c05dc61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:22:00.110800 13462 ts_tablet_manager.cc:1905] T c7dfae141a664ba780dd922f3e932998 P bcba8d834ed44491a94b32067ba681ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:22:00.111419 13463 ts_tablet_manager.cc:1905] T c7dfae141a664ba780dd922f3e932998 P b9c4150963e04e018ba397151c05dc61: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260808 22:22:00.118629 12450 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.176.193:34469 (error 108)
I20260808 22:22:00.118870 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 12803
I20260808 22:22:00.122939 13465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:00.120631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.193:34469 (local address 127.0.0.1:54450)
0808 22:22:00.122719 (+ 2088us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:00.122823 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.193:34469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1998,"thread_start_us":583,"threads_started":1}
W20260808 22:22:00.123081 12450 catalog_manager.cc:4584] TS ec50da59889f4a1e84a835ff268b75e8 (127.9.176.193:34469): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c7dfae141a664ba780dd922f3e932998: Network error: Client connection negotiation failed: client connection to 127.9.176.193:34469: connect: Connection refused (error 111)
I20260808 22:22:00.176420 13465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:00.176088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.193:34469 (local address 127.0.0.1:54466)
0808 22:22:00.176269 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:00.176337 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.193:34469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260808 22:22:00.177239 13465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:00.177010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.193:34469 (local address 127.0.0.1:54482)
0808 22:22:00.177124 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:00.177170 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.193:34469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260808 22:22:00.184012 13463 ts_tablet_manager.cc:1918] T c7dfae141a664ba780dd922f3e932998 P b9c4150963e04e018ba397151c05dc61: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 22:22:00.184100 13463 log.cc:1196] T c7dfae141a664ba780dd922f3e932998 P b9c4150963e04e018ba397151c05dc61: Deleting WAL directory at /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786227706282457-9923-0/minicluster-data/ts-2/wal/wals/c7dfae141a664ba780dd922f3e932998
I20260808 22:22:00.184325 13463 ts_tablet_manager.cc:1939] T c7dfae141a664ba780dd922f3e932998 P b9c4150963e04e018ba397151c05dc61: Deleting consensus metadata
I20260808 22:22:00.185524 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 13009
I20260808 22:22:00.190495 13465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:00.185638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.194:36839 (local address 127.0.0.1:32950)
0808 22:22:00.190319 (+ 4681us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:00.190389 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.194:36839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4584}
I20260808 22:22:00.222683 13465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:00.222337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.194:36839 (local address 127.0.0.1:32958)
0808 22:22:00.222517 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:00.222589 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.194:36839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260808 22:22:00.223385 13465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:00.223191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.194:36839 (local address 127.0.0.1:32972)
0808 22:22:00.223273 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:00.223326 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.194:36839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260808 22:22:00.235980 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 12425
2026-08-08T22:22:00Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6493 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 22:22:01 dist-test-slave-ncp9 krb5kdc[13468](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 22:22:01 dist-test-slave-ncp9 krb5kdc[13468](info): set up 2 sockets
Aug 08 22:22:01 dist-test-slave-ncp9 krb5kdc[13468](info): commencing operation
krb5kdc: starting...
W20260808 22:22:02.641870 9923 mini_kdc.cc:121] Time spent starting KDC: real 2.376s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/krb5kdc/test-user.keytab.
Aug 08 22:22:02 dist-test-slave-ncp9 krb5kdc[13468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T22:22:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:22:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.176.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.176.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.176.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab.
Entry for principal kudu/127.9.176.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab.
I20260808 22:22:02.761803 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:41639
--webserver_interface=127.9.176.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab
--principal=kudu/127.9.176.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:38055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.176.254:41639
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:22:02.870990 13490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:22:02.871227 13490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:22:02.871263 13490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:22:02.875118 13490 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:22:02.875168 13490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:22:02.875186 13490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:22:02.875200 13490 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:22:02.875213 13490 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:22:02.880291 13490 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:38055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.176.254:41639
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab
--principal=kudu/127.9.176.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:41639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.176.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13490
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:22:02.881296 13490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:22:02.882283 13490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:22:02.890295 13500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:02.890295 13498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:02.890324 13508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:22:02.890581 13490 server_base.cc:1034] running on GCE node
I20260808 22:22:02.891196 13490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:22:02.892189 13490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:22:02.893338 13490 hybrid_clock.cc:648] HybridClock initialized: now 1786227722893314 us; error 35 us; skew 500 ppm
Aug 08 22:22:02 dist-test-slave-ncp9 krb5kdc[13468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227722, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:22:02.895869 13490 init.cc:377] Logged in from keytab as kudu/127.9.176.254@KRBTEST.COM (short username kudu)
I20260808 22:22:02.896727 13490 webserver.cc:460] Webserver started at http://127.9.176.254:43015/ using document root <none> and password file <none>
I20260808 22:22:02.897271 13490 fs_manager.cc:359] Metadata directory not provided
I20260808 22:22:02.897339 13490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:22:02.897532 13490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:22:02.899497 13490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/minicluster-data/master-0/data/instance:
uuid: "66d34ca975b64fe5b785ccaea7020e75"
format_stamp: "Formatted at 2026-08-08 22:22:02 on dist-test-slave-ncp9"
server_key: "39421102b54c32ffb37fc30e36344416"
server_key_iv: "ae0343f04fce9445fbdab4d29aa879cf"
server_key_version: "encryptionkey@0"
I20260808 22:22:02.900039 13490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/minicluster-data/master-0/wal/instance:
uuid: "66d34ca975b64fe5b785ccaea7020e75"
format_stamp: "Formatted at 2026-08-08 22:22:02 on dist-test-slave-ncp9"
server_key: "39421102b54c32ffb37fc30e36344416"
server_key_iv: "ae0343f04fce9445fbdab4d29aa879cf"
server_key_version: "encryptionkey@0"
I20260808 22:22:02.903532 13490 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260808 22:22:02.905690 13516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:02.907007 13490 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 22:22:02.907159 13490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/minicluster-data/master-0/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/minicluster-data/master-0/wal
uuid: "66d34ca975b64fe5b785ccaea7020e75"
format_stamp: "Formatted at 2026-08-08 22:22:02 on dist-test-slave-ncp9"
server_key: "39421102b54c32ffb37fc30e36344416"
server_key_iv: "ae0343f04fce9445fbdab4d29aa879cf"
server_key_version: "encryptionkey@0"
I20260808 22:22:02.907260 13490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:22:02.929734 13490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:22:02.932178 13490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:22:02.932315 13490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:22:02.938830 13490 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.254:41639
I20260808 22:22:02.938845 13585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.254:41639 every 8 connection(s)
I20260808 22:22:02.939702 13490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
I20260808 22:22:02.942274 13587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:22:02.947427 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 13490
I20260808 22:22:02.947419 13587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66d34ca975b64fe5b785ccaea7020e75: Bootstrap starting.
I20260808 22:22:02.947530 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/minicluster-data/master-0/wal/instance
I20260808 22:22:02.947801 9923 external_mini_cluster.cc:1428] Setting key 13683b289f6618d59955e9241c1e6e3c
I20260808 22:22:02.949930 13587 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66d34ca975b64fe5b785ccaea7020e75: Neither blocks nor log segments found. Creating new log.
I20260808 22:22:02.950618 13587 log.cc:824] T 00000000000000000000000000000000 P 66d34ca975b64fe5b785ccaea7020e75: Log is configured to *not* fsync() on all Append() calls
I20260808 22:22:02.952476 13587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66d34ca975b64fe5b785ccaea7020e75: No bootstrap required, opened a new log
Aug 08 22:22:02 dist-test-slave-ncp9 krb5kdc[13468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
I20260808 22:22:02.956223 13587 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66d34ca975b64fe5b785ccaea7020e75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66d34ca975b64fe5b785ccaea7020e75" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 41639 } }
I20260808 22:22:02.956466 13587 raft_consensus.cc:374] T 00000000000000000000000000000000 P 66d34ca975b64fe5b785ccaea7020e75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:22:02.956542 13587 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66d34ca975b64fe5b785ccaea7020e75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66d34ca975b64fe5b785ccaea7020e75, State: Initialized, Role: FOLLOWER
I20260808 22:22:02.956998 13587 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66d34ca975b64fe5b785ccaea7020e75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66d34ca975b64fe5b785ccaea7020e75" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 41639 } }
I20260808 22:22:02.957132 13587 raft_consensus.cc:388] T 00000000000000000000000000000000 P 66d34ca975b64fe5b785ccaea7020e75 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:22:02.957195 13587 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66d34ca975b64fe5b785ccaea7020e75 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:22:02.957290 13587 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66d34ca975b64fe5b785ccaea7020e75 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:22:02.958196 13587 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66d34ca975b64fe5b785ccaea7020e75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66d34ca975b64fe5b785ccaea7020e75" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 41639 } }
I20260808 22:22:02.958500 13587 leader_election.cc:302] T 00000000000000000000000000000000 P 66d34ca975b64fe5b785ccaea7020e75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 66d34ca975b64fe5b785ccaea7020e75; no voters:
I20260808 22:22:02.958774 13587 leader_election.cc:288] T 00000000000000000000000000000000 P 66d34ca975b64fe5b785ccaea7020e75 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:22:02.958846 13595 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66d34ca975b64fe5b785ccaea7020e75 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:22:02.959070 13595 raft_consensus.cc:686] T 00000000000000000000000000000000 P 66d34ca975b64fe5b785ccaea7020e75 [term 1 LEADER]: Becoming Leader. State: Replica: 66d34ca975b64fe5b785ccaea7020e75, State: Running, Role: LEADER
I20260808 22:22:02.959362 13595 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66d34ca975b64fe5b785ccaea7020e75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66d34ca975b64fe5b785ccaea7020e75" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 41639 } }
I20260808 22:22:02.959975 13587 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66d34ca975b64fe5b785ccaea7020e75 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:22:02.960458 13595 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66d34ca975b64fe5b785ccaea7020e75 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66d34ca975b64fe5b785ccaea7020e75. Latest consensus state: current_term: 1 leader_uuid: "66d34ca975b64fe5b785ccaea7020e75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66d34ca975b64fe5b785ccaea7020e75" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 41639 } } }
I20260808 22:22:02.960565 13595 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66d34ca975b64fe5b785ccaea7020e75 [sys.catalog]: This master's current role is: LEADER
I20260808 22:22:02.961122 13598 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66d34ca975b64fe5b785ccaea7020e75 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "66d34ca975b64fe5b785ccaea7020e75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66d34ca975b64fe5b785ccaea7020e75" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 41639 } } }
I20260808 22:22:02.961237 13598 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66d34ca975b64fe5b785ccaea7020e75 [sys.catalog]: This master's current role is: LEADER
I20260808 22:22:02.961493 13601 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:22:02.961915 13591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:02.949122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34566 (local address 127.9.176.254:41639)
0808 22:22:02.949523 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:02.949537 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:02.949571 (+ 34us) server_negotiation.cc:407] Connection header received
0808 22:22:02.950232 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:02.950251 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:02.950557 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:02.950871 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:02.951676 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:02.952999 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:02.953830 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:02.954293 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:02.954448 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:02.956976 (+ 2528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:02.956981 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:02.956993 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:22:02.957018 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:02.958631 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:02.959056 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:02.959062 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:02.959068 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:02.959270 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:02.959549 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:02.959552 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:02.959553 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:02.959931 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:02.960169 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:02.960368 (+ 199us) server_negotiation.cc:299] Negotiation successful
0808 22:22:02.960792 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":134,"threads_started":1}
I20260808 22:22:02.964474 13601 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:22:02.969558 13601 catalog_manager.cc:1347] Generated new cluster ID: 887fb50f3f1f45f0b7c743ba94be452a
I20260808 22:22:02.969619 13601 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:22:02.996162 13601 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:22:02.997123 13601 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:22:03.007681 13601 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 66d34ca975b64fe5b785ccaea7020e75: Generated new TSK 0
I20260808 22:22:03.008441 13601 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.176.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.176.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.176.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.193.keytab.
Entry for principal kudu/127.9.176.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.193.keytab.
I20260808 22:22:03.043906 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.193:0
--local_ip_for_outbound_sockets=127.9.176.193
--webserver_interface=127.9.176.193
--webserver_port=0
--tserver_master_addrs=127.9.176.254:41639
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.193.keytab
--principal=kudu/127.9.176.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:38055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:22:03.151278 13628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:22:03.151495 13628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:22:03.151533 13628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:22:03.155364 13628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:22:03.155411 13628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:22:03.155463 13628 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.193
I20260808 22:22:03.160389 13628 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:38055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.193.keytab
--principal=kudu/127.9.176.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.176.193
--webserver_port=0
--tserver_master_addrs=127.9.176.254:41639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13628
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.193
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:22:03.161298 13628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:22:03.162251 13628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:22:03.171231 13636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:03.171252 13641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:22:03.171391 13628 server_base.cc:1034] running on GCE node
W20260808 22:22:03.171232 13633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:22:03.171836 13628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:22:03.172403 13628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:22:03.173549 13628 hybrid_clock.cc:648] HybridClock initialized: now 1786227723173530 us; error 37 us; skew 500 ppm
Aug 08 22:22:03 dist-test-slave-ncp9 krb5kdc[13468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227723, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:22:03.176201 13628 init.cc:377] Logged in from keytab as kudu/127.9.176.193@KRBTEST.COM (short username kudu)
I20260808 22:22:03.177013 13628 webserver.cc:460] Webserver started at http://127.9.176.193:40195/ using document root <none> and password file <none>
I20260808 22:22:03.177495 13628 fs_manager.cc:359] Metadata directory not provided
I20260808 22:22:03.177561 13628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:22:03.177747 13628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:22:03.179636 13628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-0/data/instance:
uuid: "32892b01f1594245b78820c4bda26c2b"
format_stamp: "Formatted at 2026-08-08 22:22:03 on dist-test-slave-ncp9"
server_key: "d4b78a7bb97a0225ed8a4351769ce64f"
server_key_iv: "7ac7b7c27a4f2180458112e2375a73bf"
server_key_version: "encryptionkey@0"
I20260808 22:22:03.180147 13628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-0/wal/instance:
uuid: "32892b01f1594245b78820c4bda26c2b"
format_stamp: "Formatted at 2026-08-08 22:22:03 on dist-test-slave-ncp9"
server_key: "d4b78a7bb97a0225ed8a4351769ce64f"
server_key_iv: "7ac7b7c27a4f2180458112e2375a73bf"
server_key_version: "encryptionkey@0"
I20260808 22:22:03.183684 13628 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260808 22:22:03.185838 13650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:03.186904 13628 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 22:22:03.187004 13628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-0/wal
uuid: "32892b01f1594245b78820c4bda26c2b"
format_stamp: "Formatted at 2026-08-08 22:22:03 on dist-test-slave-ncp9"
server_key: "d4b78a7bb97a0225ed8a4351769ce64f"
server_key_iv: "7ac7b7c27a4f2180458112e2375a73bf"
server_key_version: "encryptionkey@0"
I20260808 22:22:03.187096 13628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:22:03.199107 13628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:22:03.201903 13628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:22:03.202070 13628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:22:03.202672 13628 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:22:03.203433 13628 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:22:03.203490 13628 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:03.203538 13628 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:22:03.203567 13628 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:03.211767 13628 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.193:44941
I20260808 22:22:03.211818 13817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.193:44941 every 8 connection(s)
I20260808 22:22:03.212575 13628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
I20260808 22:22:03.219391 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 13628
I20260808 22:22:03.219482 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-0/wal/instance
I20260808 22:22:03.219726 9923 external_mini_cluster.cc:1428] Setting key fe9da0519350280fc7a0697b5cb6cc65
Aug 08 22:22:03 dist-test-slave-ncp9 krb5kdc[13468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227723, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.193@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
I20260808 22:22:03.225270 13591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:03.214188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.193:40457 (local address 127.9.176.254:41639)
0808 22:22:03.214304 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:03.214307 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:03.215076 (+ 769us) server_negotiation.cc:407] Connection header received
0808 22:22:03.216307 (+ 1231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:03.216311 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:03.216367 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:03.216445 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:03.217610 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:03.218138 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:03.219236 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:03.219494 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:03.219615 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:03.221895 (+ 2280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:03.221898 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:03.221901 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:22:03.221926 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:03.223321 (+ 1395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:03.223796 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:03.223800 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:03.223801 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:03.223884 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:03.224197 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:03.224201 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:03.224202 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:03.224380 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:03.224467 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:03.224970 (+ 503us) server_negotiation.cc:299] Negotiation successful
0808 22:22:03.225108 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260808 22:22:03.225980 13822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:03.214525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:41639 (local address 127.9.176.193:40457)
0808 22:22:03.214933 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:03.214972 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:03.216109 (+ 1137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:03.216614 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:03.216621 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:03.216994 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:03.217465 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:03.217478 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:03.218256 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:03.218260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:03.219117 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:03.219127 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:03.219659 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:03.219664 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:03.219833 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:03.220360 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:03.220376 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:03.221738 (+ 1362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:22:03.223430 (+ 1692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:03.223436 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:03.223450 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:03.223704 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:03.223967 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:03.223970 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:03.223972 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:03.224110 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:03.224479 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:22:03.224486 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:22:03.224719 (+ 233us) client_negotiation.cc:769] Sending connection context
0808 22:22:03.224879 (+ 160us) client_negotiation.cc:240] Negotiation successful
0808 22:22:03.225212 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":112,"threads_started":1}
I20260808 22:22:03.227074 13820 heartbeater.cc:347] Connected to a master server at 127.9.176.254:41639
I20260808 22:22:03.227298 13820 heartbeater.cc:464] Registering TS with master...
I20260808 22:22:03.227799 13820 heartbeater.cc:511] Master 127.9.176.254:41639 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.9.176.194@KRBTEST.COM; defaulting to no policy
I20260808 22:22:03.229550 13537 ts_manager.cc:194] Registered new tserver with Master: 32892b01f1594245b78820c4bda26c2b (127.9.176.193:44941)
I20260808 22:22:03.230620 13537 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.176.193@KRBTEST.COM'} at 127.9.176.193:40457
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.176.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.176.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.194.keytab.
Entry for principal kudu/127.9.176.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.194.keytab.
I20260808 22:22:03.247136 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.194:0
--local_ip_for_outbound_sockets=127.9.176.194
--webserver_interface=127.9.176.194
--webserver_port=0
--tserver_master_addrs=127.9.176.254:41639
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.194.keytab
--principal=kudu/127.9.176.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:38055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:22:03.353776 13825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:22:03.354008 13825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:22:03.354058 13825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:22:03.357918 13825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:22:03.357965 13825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:22:03.358042 13825 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.194
I20260808 22:22:03.363247 13825 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:38055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.194.keytab
--principal=kudu/127.9.176.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.176.194
--webserver_port=0
--tserver_master_addrs=127.9.176.254:41639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13825
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.194
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:22:03.364151 13825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:22:03.365118 13825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:22:03.373930 13834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:03.373952 13832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:22:03.374116 13825 server_base.cc:1034] running on GCE node
W20260808 22:22:03.373979 13836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:22:03.374537 13825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:22:03.375101 13825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:22:03.376260 13825 hybrid_clock.cc:648] HybridClock initialized: now 1786227723376234 us; error 34 us; skew 500 ppm
Aug 08 22:22:03 dist-test-slave-ncp9 krb5kdc[13468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227723, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:22:03.378846 13825 init.cc:377] Logged in from keytab as kudu/127.9.176.194@KRBTEST.COM (short username kudu)
I20260808 22:22:03.379634 13825 webserver.cc:460] Webserver started at http://127.9.176.194:34329/ using document root <none> and password file <none>
I20260808 22:22:03.380126 13825 fs_manager.cc:359] Metadata directory not provided
I20260808 22:22:03.380185 13825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:22:03.380373 13825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:22:03.382293 13825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-1/data/instance:
uuid: "17dfc0d52acb44f782f1182b54627c80"
format_stamp: "Formatted at 2026-08-08 22:22:03 on dist-test-slave-ncp9"
server_key: "1b675455e0c6a489b035f8f5efd1b64c"
server_key_iv: "102c9829ce2aef666ab84be703078e83"
server_key_version: "encryptionkey@0"
I20260808 22:22:03.382807 13825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-1/wal/instance:
uuid: "17dfc0d52acb44f782f1182b54627c80"
format_stamp: "Formatted at 2026-08-08 22:22:03 on dist-test-slave-ncp9"
server_key: "1b675455e0c6a489b035f8f5efd1b64c"
server_key_iv: "102c9829ce2aef666ab84be703078e83"
server_key_version: "encryptionkey@0"
I20260808 22:22:03.386140 13825 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 22:22:03.388243 13845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:03.389341 13825 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 22:22:03.389444 13825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-1/wal
uuid: "17dfc0d52acb44f782f1182b54627c80"
format_stamp: "Formatted at 2026-08-08 22:22:03 on dist-test-slave-ncp9"
server_key: "1b675455e0c6a489b035f8f5efd1b64c"
server_key_iv: "102c9829ce2aef666ab84be703078e83"
server_key_version: "encryptionkey@0"
I20260808 22:22:03.389539 13825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:22:03.408988 13825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:22:03.411639 13825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:22:03.411801 13825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:22:03.412326 13825 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:22:03.413116 13825 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:22:03.413163 13825 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:03.413213 13825 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:22:03.413242 13825 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:03.421356 13825 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.194:45379
I20260808 22:22:03.421393 14024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.194:45379 every 8 connection(s)
I20260808 22:22:03.422201 13825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
I20260808 22:22:03.422767 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 13825
I20260808 22:22:03.422870 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-1/wal/instance
I20260808 22:22:03.423173 9923 external_mini_cluster.cc:1428] Setting key 314d7e7fcaec8ea39a1fd2dfc5fb9c66
Aug 08 22:22:03 dist-test-slave-ncp9 krb5kdc[13468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227723, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.194@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.176.195@KRBTEST.COM; defaulting to no policy
I20260808 22:22:03.437497 13591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:03.424005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.194:35385 (local address 127.9.176.254:41639)
0808 22:22:03.424115 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:03.424119 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:03.425219 (+ 1100us) server_negotiation.cc:407] Connection header received
0808 22:22:03.426776 (+ 1557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:03.426779 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:03.426834 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:03.426916 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:03.428027 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:03.428565 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:03.430260 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:03.430590 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:03.430700 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:03.433655 (+ 2955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:03.433659 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:03.433662 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:22:03.433718 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:03.435473 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:03.436057 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:03.436061 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:03.436063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:03.436156 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:03.436487 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:03.436489 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:03.436491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:03.436679 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:03.436742 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:03.437271 (+ 529us) server_negotiation.cc:299] Negotiation successful
0808 22:22:03.437376 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260808 22:22:03.438738 14028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:03.424505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:41639 (local address 127.9.176.194:35385)
0808 22:22:03.425046 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:03.425116 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:03.426538 (+ 1422us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:03.427059 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:03.427065 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:03.427401 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:03.427887 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:03.427902 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:03.428688 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:03.428692 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:03.430099 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:03.430111 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:03.430743 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:03.430748 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:03.430949 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:03.431537 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:03.431558 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:03.433362 (+ 1804us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:22:03.435597 (+ 2235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:03.435603 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:03.435618 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:03.435941 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:03.436250 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:03.436253 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:03.436254 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:03.436402 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:03.436794 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:22:03.436799 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:22:03.437051 (+ 252us) client_negotiation.cc:769] Sending connection context
0808 22:22:03.437668 (+ 617us) client_negotiation.cc:240] Negotiation successful
0808 22:22:03.437953 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":137,"threads_started":1}
I20260808 22:22:03.439926 14025 heartbeater.cc:347] Connected to a master server at 127.9.176.254:41639
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.176.195@KRBTEST.COM" created.
I20260808 22:22:03.440629 14025 heartbeater.cc:464] Registering TS with master...
I20260808 22:22:03.441246 14025 heartbeater.cc:511] Master 127.9.176.254:41639 requested a full tablet report, sending...
I20260808 22:22:03.442446 13537 ts_manager.cc:194] Registered new tserver with Master: 17dfc0d52acb44f782f1182b54627c80 (127.9.176.194:45379)
I20260808 22:22:03.443228 13537 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.176.194@KRBTEST.COM'} at 127.9.176.194:35385
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.176.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.195.keytab.
Entry for principal kudu/127.9.176.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.195.keytab.
I20260808 22:22:03.456322 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.195:0
--local_ip_for_outbound_sockets=127.9.176.195
--webserver_interface=127.9.176.195
--webserver_port=0
--tserver_master_addrs=127.9.176.254:41639
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.195.keytab
--principal=kudu/127.9.176.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:38055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:22:03.571194 14031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:22:03.571425 14031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:22:03.571465 14031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:22:03.575397 14031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:22:03.575449 14031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:22:03.575507 14031 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.195
I20260808 22:22:03.580570 14031 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:38055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.195.keytab
--principal=kudu/127.9.176.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.176.195
--webserver_port=0
--tserver_master_addrs=127.9.176.254:41639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14031
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.195
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:22:03.581590 14031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:22:03.582571 14031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:22:03.591888 14041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:03.591939 14040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:03.592144 14043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:22:03.592186 14031 server_base.cc:1034] running on GCE node
I20260808 22:22:03.592572 14031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:22:03.593216 14031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:22:03.594398 14031 hybrid_clock.cc:648] HybridClock initialized: now 1786227723594367 us; error 40 us; skew 500 ppm
Aug 08 22:22:03 dist-test-slave-ncp9 krb5kdc[13468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227723, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:22:03.597064 14031 init.cc:377] Logged in from keytab as kudu/127.9.176.195@KRBTEST.COM (short username kudu)
I20260808 22:22:03.597887 14031 webserver.cc:460] Webserver started at http://127.9.176.195:42189/ using document root <none> and password file <none>
I20260808 22:22:03.598395 14031 fs_manager.cc:359] Metadata directory not provided
I20260808 22:22:03.598455 14031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:22:03.598664 14031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:22:03.600658 14031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-2/data/instance:
uuid: "9b273fab7f8744a3a470c4ef6ca412fa"
format_stamp: "Formatted at 2026-08-08 22:22:03 on dist-test-slave-ncp9"
server_key: "e7bb99d05ec712c240cd0333f5b262a5"
server_key_iv: "5959662618d8e8d434759c5677ab8ba9"
server_key_version: "encryptionkey@0"
I20260808 22:22:03.601205 14031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-2/wal/instance:
uuid: "9b273fab7f8744a3a470c4ef6ca412fa"
format_stamp: "Formatted at 2026-08-08 22:22:03 on dist-test-slave-ncp9"
server_key: "e7bb99d05ec712c240cd0333f5b262a5"
server_key_iv: "5959662618d8e8d434759c5677ab8ba9"
server_key_version: "encryptionkey@0"
I20260808 22:22:03.604749 14031 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260808 22:22:03.607026 14060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:03.608147 14031 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 22:22:03.608255 14031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-2/wal
uuid: "9b273fab7f8744a3a470c4ef6ca412fa"
format_stamp: "Formatted at 2026-08-08 22:22:03 on dist-test-slave-ncp9"
server_key: "e7bb99d05ec712c240cd0333f5b262a5"
server_key_iv: "5959662618d8e8d434759c5677ab8ba9"
server_key_version: "encryptionkey@0"
I20260808 22:22:03.608342 14031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:22:03.621508 14031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:22:03.624575 14031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:22:03.624742 14031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:22:03.625341 14031 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:22:03.626155 14031 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:22:03.626204 14031 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:03.626257 14031 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:22:03.626286 14031 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:03.634707 14031 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.195:42779
I20260808 22:22:03.634737 14242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.195:42779 every 8 connection(s)
I20260808 22:22:03.635569 14031 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
I20260808 22:22:03.642643 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 14031
I20260808 22:22:03.642755 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-2/wal/instance
I20260808 22:22:03.643023 9923 external_mini_cluster.cc:1428] Setting key cd91b3fa74ed38e86ae72919df98488f
Aug 08 22:22:03 dist-test-slave-ncp9 krb5kdc[13468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227723, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.195@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
I20260808 22:22:03.648809 13591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:03.637382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.195:53491 (local address 127.9.176.254:41639)
0808 22:22:03.637541 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:03.637545 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:03.638353 (+ 808us) server_negotiation.cc:407] Connection header received
0808 22:22:03.639651 (+ 1298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:03.639655 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:03.639710 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:03.639785 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:03.640966 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:03.641504 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:03.642758 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:03.643014 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:03.643120 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:03.645863 (+ 2743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:03.645866 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:03.645868 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:22:03.645891 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:03.647045 (+ 1154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:03.647517 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:03.647520 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:03.647521 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:03.647609 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:03.647898 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:03.647900 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:03.647902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:03.648064 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:03.648149 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:03.648597 (+ 448us) server_negotiation.cc:299] Negotiation successful
0808 22:22:03.648695 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 22:22:03.649609 14250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:03.637746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:41639 (local address 127.9.176.195:53491)
0808 22:22:03.638194 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:03.638235 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:03.639428 (+ 1193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:03.639964 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:03.639971 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:03.640322 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:03.640795 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:03.640810 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:03.641624 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:03.641627 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:03.642614 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:03.642627 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:03.643177 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:03.643181 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:03.643360 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:03.643977 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:03.643996 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:03.645721 (+ 1725us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:22:03.647150 (+ 1429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:03.647154 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:03.647166 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:03.647403 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:03.647697 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:03.647699 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:03.647700 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:03.647832 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:03.648147 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:22:03.648151 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:22:03.648366 (+ 215us) client_negotiation.cc:769] Sending connection context
0808 22:22:03.648542 (+ 176us) client_negotiation.cc:240] Negotiation successful
0808 22:22:03.648786 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":117,"threads_started":1}
I20260808 22:22:03.650579 14246 heartbeater.cc:347] Connected to a master server at 127.9.176.254:41639
I20260808 22:22:03.650779 14246 heartbeater.cc:464] Registering TS with master...
I20260808 22:22:03.651198 14246 heartbeater.cc:511] Master 127.9.176.254:41639 requested a full tablet report, sending...
I20260808 22:22:03.652134 13537 ts_manager.cc:194] Registered new tserver with Master: 9b273fab7f8744a3a470c4ef6ca412fa (127.9.176.195:42779)
I20260808 22:22:03.652698 13537 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.176.195@KRBTEST.COM'} at 127.9.176.195:53491
I20260808 22:22:03.656729 9923 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260808 22:22:03.670274 13591 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:22:03.669045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34570 (local address 127.9.176.254:41639)
0808 22:22:03.669170 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:03.669174 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:03.669330 (+ 156us) server_negotiation.cc:407] Connection header received
0808 22:22:03.669435 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:03.669437 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:03.669474 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:03.669548 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:03.670154 (+ 606us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34570: BlockingRecv error: recv got EOF from 127.0.0.1:34570 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 22:22:03.671371 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 13628
I20260808 22:22:03.692173 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 13825
I20260808 22:22:03.704371 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 14031
I20260808 22:22:03.714886 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 13490
2026-08-08T22:22:03Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3473 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786227706282457-9923-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 22:22:03 dist-test-slave-ncp9 krb5kdc[14271](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 22:22:03 dist-test-slave-ncp9 krb5kdc[14271](info): set up 2 sockets
Aug 08 22:22:03 dist-test-slave-ncp9 krb5kdc[14271](info): commencing operation
krb5kdc: starting...
W20260808 22:22:04.832720 9923 mini_kdc.cc:121] Time spent starting KDC: real 1.094s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786227706282457-9923-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786227706282457-9923-0/krb5kdc/test-user.keytab.
Aug 08 22:22:04 dist-test-slave-ncp9 krb5kdc[14271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227724, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T22:22:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:22:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.176.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.176.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.176.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab.
Entry for principal kudu/127.9.176.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab.
I20260808 22:22:04.947358 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786227706282457-9923-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786227706282457-9923-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786227706282457-9923-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:33073
--webserver_interface=127.9.176.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab
--principal=kudu/127.9.176.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:34929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.176.254:33073
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:22:05.056849 14288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:22:05.057114 14288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:22:05.057169 14288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:22:05.061053 14288 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:22:05.061098 14288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:22:05.061124 14288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:22:05.061143 14288 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:22:05.061164 14288 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:22:05.066136 14288 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:34929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786227706282457-9923-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786227706282457-9923-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.176.254:33073
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab
--principal=kudu/127.9.176.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:33073
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.176.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14288
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786227706282457-9923-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:22:05.067078 14288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:22:05.068022 14288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:22:05.076072 14294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:05.076076 14295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:22:05.076187 14288 server_base.cc:1034] running on GCE node
W20260808 22:22:05.076349 14299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:22:05.076845 14288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:22:05.077888 14288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:22:05.079090 14288 hybrid_clock.cc:648] HybridClock initialized: now 1786227725079061 us; error 28 us; skew 500 ppm
Aug 08 22:22:05 dist-test-slave-ncp9 krb5kdc[14271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227725, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:22:05.082054 14288 init.cc:377] Logged in from keytab as kudu/127.9.176.254@KRBTEST.COM (short username kudu)
I20260808 22:22:05.083060 14288 webserver.cc:460] Webserver started at http://127.9.176.254:44325/ using document root <none> and password file <none>
I20260808 22:22:05.083631 14288 fs_manager.cc:359] Metadata directory not provided
I20260808 22:22:05.083704 14288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:22:05.083917 14288 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:22:05.086005 14288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786227706282457-9923-0/minicluster-data/master-0/data/instance:
uuid: "b20497d1bfdd4470afeef46f7c5fc858"
format_stamp: "Formatted at 2026-08-08 22:22:05 on dist-test-slave-ncp9"
server_key: "c13e473b83b6ba2c10fb6cbed869d7b9"
server_key_iv: "f090604140ae526b8b672ae3a2ef5727"
server_key_version: "encryptionkey@0"
I20260808 22:22:05.086589 14288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786227706282457-9923-0/minicluster-data/master-0/wal/instance:
uuid: "b20497d1bfdd4470afeef46f7c5fc858"
format_stamp: "Formatted at 2026-08-08 22:22:05 on dist-test-slave-ncp9"
server_key: "c13e473b83b6ba2c10fb6cbed869d7b9"
server_key_iv: "f090604140ae526b8b672ae3a2ef5727"
server_key_version: "encryptionkey@0"
I20260808 22:22:05.090272 14288 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 22:22:05.092635 14310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:05.093848 14288 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 22:22:05.093968 14288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786227706282457-9923-0/minicluster-data/master-0/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786227706282457-9923-0/minicluster-data/master-0/wal
uuid: "b20497d1bfdd4470afeef46f7c5fc858"
format_stamp: "Formatted at 2026-08-08 22:22:05 on dist-test-slave-ncp9"
server_key: "c13e473b83b6ba2c10fb6cbed869d7b9"
server_key_iv: "f090604140ae526b8b672ae3a2ef5727"
server_key_version: "encryptionkey@0"
I20260808 22:22:05.094067 14288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786227706282457-9923-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786227706282457-9923-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786227706282457-9923-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:22:05.115571 14288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:22:05.118582 14288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:22:05.118779 14288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:22:05.126240 14288 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.254:33073
I20260808 22:22:05.126557 14395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.254:33073 every 8 connection(s)
I20260808 22:22:05.127233 14288 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
I20260808 22:22:05.129979 14396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:22:05.133104 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 14288
I20260808 22:22:05.133219 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786227706282457-9923-0/minicluster-data/master-0/wal/instance
I20260808 22:22:05.133458 9923 external_mini_cluster.cc:1428] Setting key eb146d11a99c90063ad14694f243fd93
I20260808 22:22:05.135582 14396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b20497d1bfdd4470afeef46f7c5fc858: Bootstrap starting.
I20260808 22:22:05.138018 14396 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b20497d1bfdd4470afeef46f7c5fc858: Neither blocks nor log segments found. Creating new log.
I20260808 22:22:05.138827 14396 log.cc:824] T 00000000000000000000000000000000 P b20497d1bfdd4470afeef46f7c5fc858: Log is configured to *not* fsync() on all Append() calls
I20260808 22:22:05.141736 14396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b20497d1bfdd4470afeef46f7c5fc858: No bootstrap required, opened a new log
Aug 08 22:22:05 dist-test-slave-ncp9 krb5kdc[14271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227724, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
I20260808 22:22:05.145489 14396 raft_consensus.cc:348] T 00000000000000000000000000000000 P b20497d1bfdd4470afeef46f7c5fc858 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b20497d1bfdd4470afeef46f7c5fc858" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 33073 } }
I20260808 22:22:05.145722 14396 raft_consensus.cc:374] T 00000000000000000000000000000000 P b20497d1bfdd4470afeef46f7c5fc858 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:22:05.145780 14396 raft_consensus.cc:729] T 00000000000000000000000000000000 P b20497d1bfdd4470afeef46f7c5fc858 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b20497d1bfdd4470afeef46f7c5fc858, State: Initialized, Role: FOLLOWER
I20260808 22:22:05.146251 14396 consensus_queue.cc:260] T 00000000000000000000000000000000 P b20497d1bfdd4470afeef46f7c5fc858 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b20497d1bfdd4470afeef46f7c5fc858" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 33073 } }
I20260808 22:22:05.146404 14396 raft_consensus.cc:388] T 00000000000000000000000000000000 P b20497d1bfdd4470afeef46f7c5fc858 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:22:05.146469 14396 raft_consensus.cc:482] T 00000000000000000000000000000000 P b20497d1bfdd4470afeef46f7c5fc858 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:22:05.146560 14396 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b20497d1bfdd4470afeef46f7c5fc858 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:22:05.147629 14396 raft_consensus.cc:504] T 00000000000000000000000000000000 P b20497d1bfdd4470afeef46f7c5fc858 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b20497d1bfdd4470afeef46f7c5fc858" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 33073 } }
I20260808 22:22:05.147987 14396 leader_election.cc:302] T 00000000000000000000000000000000 P b20497d1bfdd4470afeef46f7c5fc858 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b20497d1bfdd4470afeef46f7c5fc858; no voters:
I20260808 22:22:05.148298 14396 leader_election.cc:288] T 00000000000000000000000000000000 P b20497d1bfdd4470afeef46f7c5fc858 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:22:05.148346 14399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:05.134723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34246 (local address 127.9.176.254:33073)
0808 22:22:05.135118 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:05.135134 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:05.135167 (+ 33us) server_negotiation.cc:407] Connection header received
0808 22:22:05.135804 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:05.135823 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:05.136151 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:05.136463 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:05.137383 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:05.138406 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:05.139480 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:05.139961 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:05.140107 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:05.142685 (+ 2578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:05.142691 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:05.142706 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:22:05.142737 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:05.145320 (+ 2583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:05.145737 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:05.145743 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:05.145749 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:05.145891 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:05.146144 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:05.146148 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:05.146149 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:05.146652 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:05.146858 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:05.147049 (+ 191us) server_negotiation.cc:299] Negotiation successful
0808 22:22:05.147354 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":116,"threads_started":1}
I20260808 22:22:05.148396 14401 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b20497d1bfdd4470afeef46f7c5fc858 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:22:05.148769 14401 raft_consensus.cc:686] T 00000000000000000000000000000000 P b20497d1bfdd4470afeef46f7c5fc858 [term 1 LEADER]: Becoming Leader. State: Replica: b20497d1bfdd4470afeef46f7c5fc858, State: Running, Role: LEADER
I20260808 22:22:05.149122 14401 consensus_queue.cc:237] T 00000000000000000000000000000000 P b20497d1bfdd4470afeef46f7c5fc858 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b20497d1bfdd4470afeef46f7c5fc858" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 33073 } }
I20260808 22:22:05.150637 14396 sys_catalog.cc:564] T 00000000000000000000000000000000 P b20497d1bfdd4470afeef46f7c5fc858 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:22:05.150656 14403 sys_catalog.cc:455] T 00000000000000000000000000000000 P b20497d1bfdd4470afeef46f7c5fc858 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b20497d1bfdd4470afeef46f7c5fc858. Latest consensus state: current_term: 1 leader_uuid: "b20497d1bfdd4470afeef46f7c5fc858" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b20497d1bfdd4470afeef46f7c5fc858" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 33073 } } }
I20260808 22:22:05.150780 14403 sys_catalog.cc:458] T 00000000000000000000000000000000 P b20497d1bfdd4470afeef46f7c5fc858 [sys.catalog]: This master's current role is: LEADER
I20260808 22:22:05.150877 14402 sys_catalog.cc:455] T 00000000000000000000000000000000 P b20497d1bfdd4470afeef46f7c5fc858 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b20497d1bfdd4470afeef46f7c5fc858" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b20497d1bfdd4470afeef46f7c5fc858" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 33073 } } }
I20260808 22:22:05.150966 14402 sys_catalog.cc:458] T 00000000000000000000000000000000 P b20497d1bfdd4470afeef46f7c5fc858 [sys.catalog]: This master's current role is: LEADER
I20260808 22:22:05.151265 14407 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:22:05.152139 9923 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 22:22:05.154388 14407 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:22:05.160607 14407 catalog_manager.cc:1347] Generated new cluster ID: 9490846dfcde40b0a27ef117e8ee3541
I20260808 22:22:05.160742 14407 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:22:05.175432 14407 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:22:05.176182 14399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:05.170528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34260 (local address 127.9.176.254:33073)
0808 22:22:05.170652 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:05.170657 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:05.170916 (+ 259us) server_negotiation.cc:407] Connection header received
0808 22:22:05.171062 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:05.171065 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:05.171116 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:05.171193 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:05.171726 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:05.172535 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:05.173701 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:05.174061 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:05.174156 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:05.175146 (+ 990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:05.175150 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:05.175153 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:22:05.175731 (+ 578us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:05.175766 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:22:05.175786 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:05.175871 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:05.175985 (+ 114us) server_negotiation.cc:299] Negotiation successful
0808 22:22:05.176044 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 22:22:05.176488 14407 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:22:05.189314 14407 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b20497d1bfdd4470afeef46f7c5fc858: Generated new TSK 0
I20260808 22:22:05.190151 14407 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 22:22:05.203325 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 14288
2026-08-08T22:22:05Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1493 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 22:22:05 dist-test-slave-ncp9 krb5kdc[14437](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 22:22:05 dist-test-slave-ncp9 krb5kdc[14437](info): set up 2 sockets
Aug 08 22:22:05 dist-test-slave-ncp9 krb5kdc[14437](info): commencing operation
krb5kdc: starting...
W20260808 22:22:06.333509 9923 mini_kdc.cc:121] Time spent starting KDC: real 1.101s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/krb5kdc/test-user.keytab.
Aug 08 22:22:06 dist-test-slave-ncp9 krb5kdc[14437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227726, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T22:22:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:22:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.176.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.176.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.176.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab.
Entry for principal kudu/127.9.176.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab.
I20260808 22:22:06.449456 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:32937
--webserver_interface=127.9.176.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab
--principal=kudu/127.9.176.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:41935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.176.254:32937
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:22:06.555699 14455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:22:06.555948 14455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:22:06.555996 14455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:22:06.559863 14455 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:22:06.559919 14455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:22:06.559943 14455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:22:06.559964 14455 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:22:06.559983 14455 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:22:06.565078 14455 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:41935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.176.254:32937
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab
--principal=kudu/127.9.176.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:32937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.176.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14455
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:22:06.566083 14455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:22:06.567071 14455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:22:06.575291 14465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:22:06.575348 14455 server_base.cc:1034] running on GCE node
W20260808 22:22:06.575297 14469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:06.575313 14467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:22:06.576045 14455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:22:06.577150 14455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:22:06.578312 14455 hybrid_clock.cc:648] HybridClock initialized: now 1786227726578297 us; error 26 us; skew 500 ppm
Aug 08 22:22:06 dist-test-slave-ncp9 krb5kdc[14437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227726, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:22:06.583856 14455 init.cc:377] Logged in from keytab as kudu/127.9.176.254@KRBTEST.COM (short username kudu)
I20260808 22:22:06.584777 14455 webserver.cc:460] Webserver started at http://127.9.176.254:44529/ using document root <none> and password file <none>
I20260808 22:22:06.585317 14455 fs_manager.cc:359] Metadata directory not provided
I20260808 22:22:06.585386 14455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:22:06.585609 14455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:22:06.587561 14455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/minicluster-data/master-0/data/instance:
uuid: "03036a30cc1746bf998a7cef5325981a"
format_stamp: "Formatted at 2026-08-08 22:22:06 on dist-test-slave-ncp9"
server_key: "4579a6e9903d308dafcc7228286c09b7"
server_key_iv: "bbac82bca4b546dcf76455098340e1ae"
server_key_version: "encryptionkey@0"
I20260808 22:22:06.588090 14455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/minicluster-data/master-0/wal/instance:
uuid: "03036a30cc1746bf998a7cef5325981a"
format_stamp: "Formatted at 2026-08-08 22:22:06 on dist-test-slave-ncp9"
server_key: "4579a6e9903d308dafcc7228286c09b7"
server_key_iv: "bbac82bca4b546dcf76455098340e1ae"
server_key_version: "encryptionkey@0"
I20260808 22:22:06.591585 14455 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260808 22:22:06.593801 14479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:06.594929 14455 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 22:22:06.595039 14455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/minicluster-data/master-0/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/minicluster-data/master-0/wal
uuid: "03036a30cc1746bf998a7cef5325981a"
format_stamp: "Formatted at 2026-08-08 22:22:06 on dist-test-slave-ncp9"
server_key: "4579a6e9903d308dafcc7228286c09b7"
server_key_iv: "bbac82bca4b546dcf76455098340e1ae"
server_key_version: "encryptionkey@0"
I20260808 22:22:06.595131 14455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:22:06.623991 14455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:22:06.627208 14455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:22:06.627398 14455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:22:06.634361 14455 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.254:32937
I20260808 22:22:06.634611 14558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.254:32937 every 8 connection(s)
I20260808 22:22:06.635373 14455 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
I20260808 22:22:06.638129 14559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:22:06.643607 14559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03036a30cc1746bf998a7cef5325981a: Bootstrap starting.
I20260808 22:22:06.645045 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 14455
I20260808 22:22:06.645171 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/minicluster-data/master-0/wal/instance
I20260808 22:22:06.645460 9923 external_mini_cluster.cc:1428] Setting key 6f538cc3ba171aa785e658020246239d
I20260808 22:22:06.646337 14559 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 03036a30cc1746bf998a7cef5325981a: Neither blocks nor log segments found. Creating new log.
I20260808 22:22:06.647305 14559 log.cc:824] T 00000000000000000000000000000000 P 03036a30cc1746bf998a7cef5325981a: Log is configured to *not* fsync() on all Append() calls
I20260808 22:22:06.649461 14559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03036a30cc1746bf998a7cef5325981a: No bootstrap required, opened a new log
I20260808 22:22:06.653519 14559 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03036a30cc1746bf998a7cef5325981a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03036a30cc1746bf998a7cef5325981a" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 32937 } }
I20260808 22:22:06.653748 14559 raft_consensus.cc:374] T 00000000000000000000000000000000 P 03036a30cc1746bf998a7cef5325981a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:22:06.653806 14559 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03036a30cc1746bf998a7cef5325981a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03036a30cc1746bf998a7cef5325981a, State: Initialized, Role: FOLLOWER
I20260808 22:22:06.654243 14559 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03036a30cc1746bf998a7cef5325981a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03036a30cc1746bf998a7cef5325981a" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 32937 } }
I20260808 22:22:06.654404 14559 raft_consensus.cc:388] T 00000000000000000000000000000000 P 03036a30cc1746bf998a7cef5325981a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:22:06.654474 14559 raft_consensus.cc:482] T 00000000000000000000000000000000 P 03036a30cc1746bf998a7cef5325981a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:22:06.654575 14559 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03036a30cc1746bf998a7cef5325981a [term 0 FOLLOWER]: Advancing to term 1
Aug 08 22:22:06 dist-test-slave-ncp9 krb5kdc[14437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227726, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
I20260808 22:22:06.655519 14559 raft_consensus.cc:504] T 00000000000000000000000000000000 P 03036a30cc1746bf998a7cef5325981a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03036a30cc1746bf998a7cef5325981a" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 32937 } }
I20260808 22:22:06.655890 14559 leader_election.cc:302] T 00000000000000000000000000000000 P 03036a30cc1746bf998a7cef5325981a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 03036a30cc1746bf998a7cef5325981a; no voters:
I20260808 22:22:06.656231 14559 leader_election.cc:288] T 00000000000000000000000000000000 P 03036a30cc1746bf998a7cef5325981a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:22:06.656327 14567 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 03036a30cc1746bf998a7cef5325981a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:22:06.656569 14567 raft_consensus.cc:686] T 00000000000000000000000000000000 P 03036a30cc1746bf998a7cef5325981a [term 1 LEADER]: Becoming Leader. State: Replica: 03036a30cc1746bf998a7cef5325981a, State: Running, Role: LEADER
I20260808 22:22:06.656949 14567 consensus_queue.cc:237] T 00000000000000000000000000000000 P 03036a30cc1746bf998a7cef5325981a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03036a30cc1746bf998a7cef5325981a" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 32937 } }
I20260808 22:22:06.657553 14559 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03036a30cc1746bf998a7cef5325981a [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:22:06.658120 14570 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03036a30cc1746bf998a7cef5325981a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03036a30cc1746bf998a7cef5325981a. Latest consensus state: current_term: 1 leader_uuid: "03036a30cc1746bf998a7cef5325981a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03036a30cc1746bf998a7cef5325981a" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 32937 } } }
I20260808 22:22:06.658244 14570 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03036a30cc1746bf998a7cef5325981a [sys.catalog]: This master's current role is: LEADER
I20260808 22:22:06.658880 14569 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03036a30cc1746bf998a7cef5325981a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "03036a30cc1746bf998a7cef5325981a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03036a30cc1746bf998a7cef5325981a" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 32937 } } }
I20260808 22:22:06.659053 14569 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03036a30cc1746bf998a7cef5325981a [sys.catalog]: This master's current role is: LEADER
I20260808 22:22:06.659955 14575 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:22:06.661019 14563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:06.646685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49600 (local address 127.9.176.254:32937)
0808 22:22:06.647092 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:06.647111 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:06.647164 (+ 53us) server_negotiation.cc:407] Connection header received
0808 22:22:06.647945 (+ 781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:06.647966 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:06.648367 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:06.648717 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:06.649449 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:06.650939 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:06.651796 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:06.652537 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:06.652702 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:06.655252 (+ 2550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:06.655259 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:06.655275 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:22:06.655305 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:06.657240 (+ 1935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:06.657753 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:06.657759 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:06.657765 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:06.657899 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:06.658255 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:06.658259 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:06.658260 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:06.658795 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:06.658990 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:06.659749 (+ 759us) server_negotiation.cc:299] Negotiation successful
0808 22:22:06.660047 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":118,"threads_started":1}
I20260808 22:22:06.662760 14575 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:22:06.667725 14575 catalog_manager.cc:1347] Generated new cluster ID: 1607e96be9244406bfc296b36c7046f2
I20260808 22:22:06.667785 14575 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:22:06.679553 14575 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:22:06.680737 14575 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:22:06.687132 14575 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 03036a30cc1746bf998a7cef5325981a: Generated new TSK 0
I20260808 22:22:06.687878 14575 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.176.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.176.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.176.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.193.keytab.
Entry for principal kudu/127.9.176.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.193.keytab.
I20260808 22:22:06.715914 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.193:0
--local_ip_for_outbound_sockets=127.9.176.193
--webserver_interface=127.9.176.193
--webserver_port=0
--tserver_master_addrs=127.9.176.254:32937
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.193.keytab
--principal=kudu/127.9.176.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:41935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:22:06.829236 14597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:22:06.829473 14597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:22:06.829512 14597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:22:06.833387 14597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:22:06.833436 14597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:22:06.833495 14597 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.193
I20260808 22:22:06.838510 14597 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:41935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.193.keytab
--principal=kudu/127.9.176.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.176.193
--webserver_port=0
--tserver_master_addrs=127.9.176.254:32937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14597
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.193
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:22:06.839519 14597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:22:06.840467 14597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:22:06.849464 14605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:06.849465 14608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:06.849488 14606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:22:06.849778 14597 server_base.cc:1034] running on GCE node
I20260808 22:22:06.850133 14597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:22:06.850692 14597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:22:06.851830 14597 hybrid_clock.cc:648] HybridClock initialized: now 1786227726851808 us; error 39 us; skew 500 ppm
Aug 08 22:22:06 dist-test-slave-ncp9 krb5kdc[14437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227726, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:22:06.854398 14597 init.cc:377] Logged in from keytab as kudu/127.9.176.193@KRBTEST.COM (short username kudu)
I20260808 22:22:06.855191 14597 webserver.cc:460] Webserver started at http://127.9.176.193:37639/ using document root <none> and password file <none>
I20260808 22:22:06.855695 14597 fs_manager.cc:359] Metadata directory not provided
I20260808 22:22:06.855754 14597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:22:06.855930 14597 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:22:06.857951 14597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-0/data/instance:
uuid: "d7dc89f7472845cca94f2adc103a1b2a"
format_stamp: "Formatted at 2026-08-08 22:22:06 on dist-test-slave-ncp9"
server_key: "ff6894658b7190934b1b6070402b8425"
server_key_iv: "a133d0fd384a6d28cc8ed32a9ff6f226"
server_key_version: "encryptionkey@0"
I20260808 22:22:06.858533 14597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-0/wal/instance:
uuid: "d7dc89f7472845cca94f2adc103a1b2a"
format_stamp: "Formatted at 2026-08-08 22:22:06 on dist-test-slave-ncp9"
server_key: "ff6894658b7190934b1b6070402b8425"
server_key_iv: "a133d0fd384a6d28cc8ed32a9ff6f226"
server_key_version: "encryptionkey@0"
I20260808 22:22:06.862087 14597 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 22:22:06.864313 14615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:06.865448 14597 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 22:22:06.865545 14597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-0/wal
uuid: "d7dc89f7472845cca94f2adc103a1b2a"
format_stamp: "Formatted at 2026-08-08 22:22:06 on dist-test-slave-ncp9"
server_key: "ff6894658b7190934b1b6070402b8425"
server_key_iv: "a133d0fd384a6d28cc8ed32a9ff6f226"
server_key_version: "encryptionkey@0"
I20260808 22:22:06.865639 14597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:22:06.880669 14597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:22:06.883358 14597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:22:06.883517 14597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:22:06.884049 14597 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:22:06.884806 14597 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:22:06.884851 14597 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:06.884915 14597 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:22:06.884948 14597 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:06.893308 14597 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.193:38915
I20260808 22:22:06.893332 14780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.193:38915 every 8 connection(s)
I20260808 22:22:06.894134 14597 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
I20260808 22:22:06.902338 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 14597
I20260808 22:22:06.902465 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-0/wal/instance
I20260808 22:22:06.902804 9923 external_mini_cluster.cc:1428] Setting key d542be4fa15bbab961314a5a6a01ae0f
Aug 08 22:22:06 dist-test-slave-ncp9 krb5kdc[14437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227726, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.193@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
I20260808 22:22:06.907436 14563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:06.895817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.193:48533 (local address 127.9.176.254:32937)
0808 22:22:06.895939 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:06.895942 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:06.896715 (+ 773us) server_negotiation.cc:407] Connection header received
0808 22:22:06.898180 (+ 1465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:06.898184 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:06.898240 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:06.898314 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:06.899487 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:06.900032 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:06.901200 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:06.901434 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:06.901524 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:06.904065 (+ 2541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:06.904069 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:06.904072 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:22:06.904097 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:06.905538 (+ 1441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:06.905986 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:06.905989 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:06.905991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:06.906082 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:06.906443 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:06.906445 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:06.906446 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:06.906612 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:06.906689 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:06.907224 (+ 535us) server_negotiation.cc:299] Negotiation successful
0808 22:22:06.907320 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 22:22:06.908452 14783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:06.896159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:32937 (local address 127.9.176.193:48533)
0808 22:22:06.896577 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:06.896617 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:06.897979 (+ 1362us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:06.898493 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:06.898499 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:06.898842 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:06.899345 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:06.899358 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:06.900160 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:06.900163 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:06.901085 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:06.901094 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:06.901571 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:06.901576 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:06.901733 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:06.902286 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:06.902307 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:06.903904 (+ 1597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:22:06.905646 (+ 1742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:06.905650 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:06.905661 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:06.905900 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:06.906173 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:06.906176 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:06.906178 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:06.906357 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:06.906700 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:22:06.906706 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:22:06.906995 (+ 289us) client_negotiation.cc:769] Sending connection context
0808 22:22:06.907195 (+ 200us) client_negotiation.cc:240] Negotiation successful
0808 22:22:06.907511 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":116,"threads_started":1}
I20260808 22:22:06.909684 14781 heartbeater.cc:347] Connected to a master server at 127.9.176.254:32937
I20260808 22:22:06.909900 14781 heartbeater.cc:464] Registering TS with master...
I20260808 22:22:06.910360 14781 heartbeater.cc:511] Master 127.9.176.254:32937 requested a full tablet report, sending...
I20260808 22:22:06.911791 14496 ts_manager.cc:194] Registered new tserver with Master: d7dc89f7472845cca94f2adc103a1b2a (127.9.176.193:38915)
I20260808 22:22:06.912863 14496 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.176.193@KRBTEST.COM'} at 127.9.176.193:48533
WARNING: no policy specified for kudu/127.9.176.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.176.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.176.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.194.keytab.
Entry for principal kudu/127.9.176.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.194.keytab.
I20260808 22:22:06.929378 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.194:0
--local_ip_for_outbound_sockets=127.9.176.194
--webserver_interface=127.9.176.194
--webserver_port=0
--tserver_master_addrs=127.9.176.254:32937
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.194.keytab
--principal=kudu/127.9.176.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:41935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:22:07.037233 14786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:22:07.037467 14786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:22:07.037524 14786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:22:07.041440 14786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:22:07.041494 14786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:22:07.041569 14786 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.194
I20260808 22:22:07.046775 14786 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:41935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.194.keytab
--principal=kudu/127.9.176.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.176.194
--webserver_port=0
--tserver_master_addrs=127.9.176.254:32937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14786
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.194
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:22:07.047725 14786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:22:07.048657 14786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:22:07.057693 14793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:07.057778 14794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:07.057701 14796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:22:07.057989 14786 server_base.cc:1034] running on GCE node
I20260808 22:22:07.058355 14786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:22:07.058985 14786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:22:07.060139 14786 hybrid_clock.cc:648] HybridClock initialized: now 1786227727060125 us; error 28 us; skew 500 ppm
Aug 08 22:22:07 dist-test-slave-ncp9 krb5kdc[14437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227727, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:22:07.062681 14786 init.cc:377] Logged in from keytab as kudu/127.9.176.194@KRBTEST.COM (short username kudu)
I20260808 22:22:07.063470 14786 webserver.cc:460] Webserver started at http://127.9.176.194:33277/ using document root <none> and password file <none>
I20260808 22:22:07.063967 14786 fs_manager.cc:359] Metadata directory not provided
I20260808 22:22:07.064033 14786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:22:07.064234 14786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:22:07.066184 14786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-1/data/instance:
uuid: "c71f1820a30a443fb6e36b42569529c3"
format_stamp: "Formatted at 2026-08-08 22:22:07 on dist-test-slave-ncp9"
server_key: "fed287f176369b85fafb3b6ed6daf89f"
server_key_iv: "b1d5b7162ab9c29bcfc496d4cb9bfef4"
server_key_version: "encryptionkey@0"
I20260808 22:22:07.066697 14786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-1/wal/instance:
uuid: "c71f1820a30a443fb6e36b42569529c3"
format_stamp: "Formatted at 2026-08-08 22:22:07 on dist-test-slave-ncp9"
server_key: "fed287f176369b85fafb3b6ed6daf89f"
server_key_iv: "b1d5b7162ab9c29bcfc496d4cb9bfef4"
server_key_version: "encryptionkey@0"
I20260808 22:22:07.070096 14786 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 22:22:07.072364 14805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:07.073565 14786 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 22:22:07.073684 14786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-1/wal
uuid: "c71f1820a30a443fb6e36b42569529c3"
format_stamp: "Formatted at 2026-08-08 22:22:07 on dist-test-slave-ncp9"
server_key: "fed287f176369b85fafb3b6ed6daf89f"
server_key_iv: "b1d5b7162ab9c29bcfc496d4cb9bfef4"
server_key_version: "encryptionkey@0"
I20260808 22:22:07.073777 14786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:22:07.098012 14786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:22:07.101178 14786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:22:07.101361 14786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:22:07.101965 14786 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:22:07.102824 14786 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:22:07.102873 14786 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:07.102926 14786 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:22:07.102953 14786 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:07.111505 14786 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.194:40039
I20260808 22:22:07.111536 14979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.194:40039 every 8 connection(s)
I20260808 22:22:07.112380 14786 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
I20260808 22:22:07.115211 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 14786
I20260808 22:22:07.115343 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-1/wal/instance
I20260808 22:22:07.115679 9923 external_mini_cluster.cc:1428] Setting key d4f8addb5c1cb1afd0d11144fcf0d2b5
Aug 08 22:22:07 dist-test-slave-ncp9 krb5kdc[14437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227727, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.194@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.176.195@KRBTEST.COM; defaulting to no policy
I20260808 22:22:07.126261 14563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:07.114159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.194:55881 (local address 127.9.176.254:32937)
0808 22:22:07.114298 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:07.114301 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:07.115158 (+ 857us) server_negotiation.cc:407] Connection header received
0808 22:22:07.116773 (+ 1615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:07.116776 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:07.116823 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:07.116921 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:07.118414 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:07.118927 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:07.120115 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:07.120385 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:07.120490 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:07.123005 (+ 2515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:07.123008 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:07.123010 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:22:07.123035 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:07.124359 (+ 1324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:07.124820 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:07.124823 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:07.124825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:07.124926 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:07.125243 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:07.125246 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:07.125248 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:07.125417 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:07.125501 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:07.125987 (+ 486us) server_negotiation.cc:299] Negotiation successful
0808 22:22:07.126104 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 22:22:07.127110 14982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:07.114540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:32937 (local address 127.9.176.194:55881)
0808 22:22:07.114980 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:07.115038 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:07.116571 (+ 1533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:07.117112 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:07.117122 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:07.117607 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:07.118257 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:07.118275 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:07.119058 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:07.119062 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:07.119996 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:07.120006 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:07.120500 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:07.120502 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:07.120643 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:07.121219 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:07.121234 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:07.122854 (+ 1620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:22:07.124467 (+ 1613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:07.124473 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:07.124486 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:07.124729 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:07.125015 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:07.125025 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:07.125027 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:07.125158 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:07.125515 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:22:07.125522 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:22:07.125757 (+ 235us) client_negotiation.cc:769] Sending connection context
0808 22:22:07.125929 (+ 172us) client_negotiation.cc:240] Negotiation successful
0808 22:22:07.126238 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":117,"threads_started":1}
I20260808 22:22:07.128165 14980 heartbeater.cc:347] Connected to a master server at 127.9.176.254:32937
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.176.195@KRBTEST.COM" created.
I20260808 22:22:07.128400 14980 heartbeater.cc:464] Registering TS with master...
I20260808 22:22:07.128862 14980 heartbeater.cc:511] Master 127.9.176.254:32937 requested a full tablet report, sending...
I20260808 22:22:07.129902 14496 ts_manager.cc:194] Registered new tserver with Master: c71f1820a30a443fb6e36b42569529c3 (127.9.176.194:40039)
I20260808 22:22:07.130620 14496 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.176.194@KRBTEST.COM'} at 127.9.176.194:55881
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.176.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.195.keytab.
Entry for principal kudu/127.9.176.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.195.keytab.
I20260808 22:22:07.141531 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.195:0
--local_ip_for_outbound_sockets=127.9.176.195
--webserver_interface=127.9.176.195
--webserver_port=0
--tserver_master_addrs=127.9.176.254:32937
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.195.keytab
--principal=kudu/127.9.176.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:41935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:22:07.249161 14986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:22:07.249384 14986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:22:07.249423 14986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:22:07.253262 14986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:22:07.253309 14986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:22:07.253362 14986 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.195
I20260808 22:22:07.258437 14986 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:41935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.195.keytab
--principal=kudu/127.9.176.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.176.195
--webserver_port=0
--tserver_master_addrs=127.9.176.254:32937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14986
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.195
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:22:07.259390 14986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:22:07.260345 14986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:22:07.269382 14994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:07.269389 14995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:07.269425 14998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:22:07.269668 14986 server_base.cc:1034] running on GCE node
I20260808 22:22:07.270068 14986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:22:07.270659 14986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:22:07.271811 14986 hybrid_clock.cc:648] HybridClock initialized: now 1786227727271785 us; error 37 us; skew 500 ppm
Aug 08 22:22:07 dist-test-slave-ncp9 krb5kdc[14437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227727, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:22:07.274410 14986 init.cc:377] Logged in from keytab as kudu/127.9.176.195@KRBTEST.COM (short username kudu)
I20260808 22:22:07.275285 14986 webserver.cc:460] Webserver started at http://127.9.176.195:34873/ using document root <none> and password file <none>
I20260808 22:22:07.275795 14986 fs_manager.cc:359] Metadata directory not provided
I20260808 22:22:07.275861 14986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:22:07.276067 14986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:22:07.278190 14986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-2/data/instance:
uuid: "7f0b8e706749422db932b59ca7289e36"
format_stamp: "Formatted at 2026-08-08 22:22:07 on dist-test-slave-ncp9"
server_key: "b46b329692f928affe45735f744e0a5a"
server_key_iv: "d8dcf909a77e5071bbb44d3b4128cf66"
server_key_version: "encryptionkey@0"
I20260808 22:22:07.278771 14986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-2/wal/instance:
uuid: "7f0b8e706749422db932b59ca7289e36"
format_stamp: "Formatted at 2026-08-08 22:22:07 on dist-test-slave-ncp9"
server_key: "b46b329692f928affe45735f744e0a5a"
server_key_iv: "d8dcf909a77e5071bbb44d3b4128cf66"
server_key_version: "encryptionkey@0"
I20260808 22:22:07.282361 14986 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 22:22:07.284629 15007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:07.285801 14986 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 22:22:07.285912 14986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-2/wal
uuid: "7f0b8e706749422db932b59ca7289e36"
format_stamp: "Formatted at 2026-08-08 22:22:07 on dist-test-slave-ncp9"
server_key: "b46b329692f928affe45735f744e0a5a"
server_key_iv: "d8dcf909a77e5071bbb44d3b4128cf66"
server_key_version: "encryptionkey@0"
I20260808 22:22:07.286016 14986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:22:07.297008 14986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:22:07.299551 14986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:22:07.299706 14986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:22:07.300210 14986 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:22:07.300994 14986 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:22:07.301048 14986 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:07.301101 14986 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:22:07.301131 14986 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:07.309042 14986 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.195:36783
I20260808 22:22:07.309096 15176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.195:36783 every 8 connection(s)
I20260808 22:22:07.309851 14986 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
I20260808 22:22:07.316859 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 14986
I20260808 22:22:07.316978 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786227706282457-9923-0/minicluster-data/ts-2/wal/instance
I20260808 22:22:07.317255 9923 external_mini_cluster.cc:1428] Setting key 9e4118bcb8d30285d46f59755e642070
Aug 08 22:22:07 dist-test-slave-ncp9 krb5kdc[14437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227727, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.195@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
I20260808 22:22:07.322202 14563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:07.311524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.195:40733 (local address 127.9.176.254:32937)
0808 22:22:07.311645 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:07.311648 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:07.312390 (+ 742us) server_negotiation.cc:407] Connection header received
0808 22:22:07.313700 (+ 1310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:07.313703 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:07.313750 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:07.313810 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:07.314871 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:07.315389 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:07.316409 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:07.316626 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:07.316711 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:07.319038 (+ 2327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:07.319041 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:07.319043 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:22:07.319066 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:07.320236 (+ 1170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:07.320693 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:07.320696 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:07.320697 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:07.320814 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:07.321140 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:07.321144 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:07.321146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:07.321352 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:07.321439 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:07.321910 (+ 471us) server_negotiation.cc:299] Negotiation successful
0808 22:22:07.322037 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260808 22:22:07.322871 15180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:07.311828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:32937 (local address 127.9.176.195:40733)
0808 22:22:07.312251 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:07.312291 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:07.313502 (+ 1211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:07.313953 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:07.313960 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:07.314297 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:07.314741 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:07.314754 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:07.315477 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:07.315479 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:07.316320 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:07.316329 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:07.316765 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:07.316769 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:07.316971 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:07.317542 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:07.317557 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:07.318886 (+ 1329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:22:07.320339 (+ 1453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:07.320345 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:07.320358 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:07.320602 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:07.320917 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:07.320920 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:07.320922 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:07.321052 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:07.321453 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:22:07.321467 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:22:07.321682 (+ 215us) client_negotiation.cc:769] Sending connection context
0808 22:22:07.321830 (+ 148us) client_negotiation.cc:240] Negotiation successful
0808 22:22:07.322118 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":127,"threads_started":1}
I20260808 22:22:07.323846 15177 heartbeater.cc:347] Connected to a master server at 127.9.176.254:32937
I20260808 22:22:07.324066 15177 heartbeater.cc:464] Registering TS with master...
I20260808 22:22:07.324496 15177 heartbeater.cc:511] Master 127.9.176.254:32937 requested a full tablet report, sending...
I20260808 22:22:07.325431 14496 ts_manager.cc:194] Registered new tserver with Master: 7f0b8e706749422db932b59ca7289e36 (127.9.176.195:36783)
I20260808 22:22:07.325963 14496 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.176.195@KRBTEST.COM'} at 127.9.176.195:40733
I20260808 22:22:07.330631 9923 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 22:22:07 dist-test-slave-ncp9 krb5kdc[14437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227727, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 08 22:22:07 dist-test-slave-ncp9 krb5kdc[14437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227727, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
I20260808 22:22:07.354838 14563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:07.347443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49614 (local address 127.9.176.254:32937)
0808 22:22:07.347550 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:07.347554 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:07.347730 (+ 176us) server_negotiation.cc:407] Connection header received
0808 22:22:07.347851 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:07.347853 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:07.347891 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:07.347949 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:07.348549 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:07.349114 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:07.349909 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:07.350132 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:07.350213 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:07.352381 (+ 2168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:07.352385 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:07.352386 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:22:07.352410 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:07.353605 (+ 1195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:07.353957 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:07.353959 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:07.353960 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:07.354033 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:07.354207 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:07.354209 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:07.354210 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:07.354397 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:07.354493 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:07.354636 (+ 143us) server_negotiation.cc:299] Negotiation successful
0808 22:22:07.354723 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260808 22:22:07.355201 14496 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:49614
I20260808 22:22:07.355934 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 14597
I20260808 22:22:07.366076 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 14786
I20260808 22:22:07.376426 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 14986
I20260808 22:22:07.395030 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 14455
2026-08-08T22:22:07Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2186 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 22:22:07 dist-test-slave-ncp9 krb5kdc[15193](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 22:22:07 dist-test-slave-ncp9 krb5kdc[15193](info): set up 2 sockets
Aug 08 22:22:07 dist-test-slave-ncp9 krb5kdc[15193](info): commencing operation
krb5kdc: starting...
W20260808 22:22:08.524679 9923 mini_kdc.cc:121] Time spent starting KDC: real 1.106s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/krb5kdc/test-user.keytab.
Aug 08 22:22:08 dist-test-slave-ncp9 krb5kdc[15193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227728, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T22:22:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:22:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.176.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.176.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.176.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab.
Entry for principal kudu/127.9.176.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab.
I20260808 22:22:08.643718 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:43673
--webserver_interface=127.9.176.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab
--principal=kudu/127.9.176.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:36489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.176.254:43673
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:22:08.754216 15210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:22:08.754465 15210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:22:08.754503 15210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:22:08.758332 15210 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:22:08.758373 15210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:22:08.758391 15210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:22:08.758406 15210 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:22:08.758420 15210 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:22:08.763489 15210 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:36489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.176.254:43673
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab
--principal=kudu/127.9.176.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:43673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.176.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15210
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:22:08.764458 15210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:22:08.765436 15210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:22:08.773903 15225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:08.773903 15221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:08.773990 15227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:22:08.774214 15210 server_base.cc:1034] running on GCE node
I20260808 22:22:08.774765 15210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:22:08.775800 15210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:22:08.776976 15210 hybrid_clock.cc:648] HybridClock initialized: now 1786227728776956 us; error 29 us; skew 500 ppm
Aug 08 22:22:08 dist-test-slave-ncp9 krb5kdc[15193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227728, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:22:08.779806 15210 init.cc:377] Logged in from keytab as kudu/127.9.176.254@KRBTEST.COM (short username kudu)
I20260808 22:22:08.780608 15210 webserver.cc:460] Webserver started at http://127.9.176.254:38493/ using document root <none> and password file <none>
I20260808 22:22:08.781158 15210 fs_manager.cc:359] Metadata directory not provided
I20260808 22:22:08.781220 15210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:22:08.781383 15210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:22:08.783319 15210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/minicluster-data/master-0/data/instance:
uuid: "85cd29cd18fe48368a7908ea28dcbaf4"
format_stamp: "Formatted at 2026-08-08 22:22:08 on dist-test-slave-ncp9"
server_key: "0e7fcca50b2fab3e1e99d632f9b97356"
server_key_iv: "e414c9cb03eb79a65ffdf26a5a5d092e"
server_key_version: "encryptionkey@0"
I20260808 22:22:08.783834 15210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/minicluster-data/master-0/wal/instance:
uuid: "85cd29cd18fe48368a7908ea28dcbaf4"
format_stamp: "Formatted at 2026-08-08 22:22:08 on dist-test-slave-ncp9"
server_key: "0e7fcca50b2fab3e1e99d632f9b97356"
server_key_iv: "e414c9cb03eb79a65ffdf26a5a5d092e"
server_key_version: "encryptionkey@0"
I20260808 22:22:08.787289 15210 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 22:22:08.789554 15237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:08.790674 15210 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 22:22:08.790779 15210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/minicluster-data/master-0/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/minicluster-data/master-0/wal
uuid: "85cd29cd18fe48368a7908ea28dcbaf4"
format_stamp: "Formatted at 2026-08-08 22:22:08 on dist-test-slave-ncp9"
server_key: "0e7fcca50b2fab3e1e99d632f9b97356"
server_key_iv: "e414c9cb03eb79a65ffdf26a5a5d092e"
server_key_version: "encryptionkey@0"
I20260808 22:22:08.790874 15210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:22:08.804430 15210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:22:08.807032 15210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:22:08.807205 15210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:22:08.813818 15210 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.254:43673
I20260808 22:22:08.813838 15316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.254:43673 every 8 connection(s)
I20260808 22:22:08.814729 15210 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
I20260808 22:22:08.817420 15318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:22:08.819772 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 15210
I20260808 22:22:08.819867 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/minicluster-data/master-0/wal/instance
I20260808 22:22:08.820115 9923 external_mini_cluster.cc:1428] Setting key 2455e68f2105811434b3fc18d393597c
I20260808 22:22:08.823173 15318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85cd29cd18fe48368a7908ea28dcbaf4: Bootstrap starting.
I20260808 22:22:08.825708 15318 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85cd29cd18fe48368a7908ea28dcbaf4: Neither blocks nor log segments found. Creating new log.
I20260808 22:22:08.826529 15318 log.cc:824] T 00000000000000000000000000000000 P 85cd29cd18fe48368a7908ea28dcbaf4: Log is configured to *not* fsync() on all Append() calls
I20260808 22:22:08.828334 15318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85cd29cd18fe48368a7908ea28dcbaf4: No bootstrap required, opened a new log
Aug 08 22:22:08 dist-test-slave-ncp9 krb5kdc[15193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227728, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
I20260808 22:22:08.832116 15318 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85cd29cd18fe48368a7908ea28dcbaf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85cd29cd18fe48368a7908ea28dcbaf4" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 43673 } }
I20260808 22:22:08.832401 15318 raft_consensus.cc:374] T 00000000000000000000000000000000 P 85cd29cd18fe48368a7908ea28dcbaf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:22:08.832492 15318 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85cd29cd18fe48368a7908ea28dcbaf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85cd29cd18fe48368a7908ea28dcbaf4, State: Initialized, Role: FOLLOWER
I20260808 22:22:08.832998 15318 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85cd29cd18fe48368a7908ea28dcbaf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85cd29cd18fe48368a7908ea28dcbaf4" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 43673 } }
I20260808 22:22:08.833177 15318 raft_consensus.cc:388] T 00000000000000000000000000000000 P 85cd29cd18fe48368a7908ea28dcbaf4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:22:08.833280 15318 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85cd29cd18fe48368a7908ea28dcbaf4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:22:08.833389 15318 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85cd29cd18fe48368a7908ea28dcbaf4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:22:08.834482 15318 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85cd29cd18fe48368a7908ea28dcbaf4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85cd29cd18fe48368a7908ea28dcbaf4" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 43673 } }
I20260808 22:22:08.834729 15322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:08.821389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35138 (local address 127.9.176.254:43673)
0808 22:22:08.821776 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:08.821792 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:08.821827 (+ 35us) server_negotiation.cc:407] Connection header received
0808 22:22:08.822469 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:08.822486 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:08.822810 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:08.823077 (+ 267us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:08.823971 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:08.825016 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:08.826144 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:08.826603 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:08.826747 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:08.829155 (+ 2408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:08.829161 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:08.829178 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:22:08.829209 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:08.831636 (+ 2427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:08.832055 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:08.832061 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:08.832066 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:08.832219 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:08.832452 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:08.832455 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:08.832457 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:08.832986 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:08.833202 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:08.833454 (+ 252us) server_negotiation.cc:299] Negotiation successful
0808 22:22:08.833759 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":104,"threads_started":1}
I20260808 22:22:08.834838 15318 leader_election.cc:302] T 00000000000000000000000000000000 P 85cd29cd18fe48368a7908ea28dcbaf4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85cd29cd18fe48368a7908ea28dcbaf4; no voters:
I20260808 22:22:08.835140 15318 leader_election.cc:288] T 00000000000000000000000000000000 P 85cd29cd18fe48368a7908ea28dcbaf4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:22:08.835251 15325 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85cd29cd18fe48368a7908ea28dcbaf4 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:22:08.835464 15325 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85cd29cd18fe48368a7908ea28dcbaf4 [term 1 LEADER]: Becoming Leader. State: Replica: 85cd29cd18fe48368a7908ea28dcbaf4, State: Running, Role: LEADER
I20260808 22:22:08.835805 15325 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85cd29cd18fe48368a7908ea28dcbaf4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85cd29cd18fe48368a7908ea28dcbaf4" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 43673 } }
I20260808 22:22:08.836376 15318 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85cd29cd18fe48368a7908ea28dcbaf4 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:22:08.836853 15325 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85cd29cd18fe48368a7908ea28dcbaf4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85cd29cd18fe48368a7908ea28dcbaf4. Latest consensus state: current_term: 1 leader_uuid: "85cd29cd18fe48368a7908ea28dcbaf4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85cd29cd18fe48368a7908ea28dcbaf4" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 43673 } } }
I20260808 22:22:08.837069 15325 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85cd29cd18fe48368a7908ea28dcbaf4 [sys.catalog]: This master's current role is: LEADER
I20260808 22:22:08.837436 15326 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85cd29cd18fe48368a7908ea28dcbaf4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "85cd29cd18fe48368a7908ea28dcbaf4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85cd29cd18fe48368a7908ea28dcbaf4" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 43673 } } }
I20260808 22:22:08.837610 15326 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85cd29cd18fe48368a7908ea28dcbaf4 [sys.catalog]: This master's current role is: LEADER
I20260808 22:22:08.837489 15332 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:22:08.840741 15332 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:22:08.846557 15332 catalog_manager.cc:1347] Generated new cluster ID: 2031c23d09f64da49bf315e71ca5e0b9
I20260808 22:22:08.846621 15332 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:22:08.852036 15332 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:22:08.852838 15332 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:22:08.861454 15332 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 85cd29cd18fe48368a7908ea28dcbaf4: Generated new TSK 0
I20260808 22:22:08.862063 15332 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.176.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.176.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.176.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.193.keytab.
Entry for principal kudu/127.9.176.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.193.keytab.
I20260808 22:22:08.895335 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.193:0
--local_ip_for_outbound_sockets=127.9.176.193
--webserver_interface=127.9.176.193
--webserver_port=0
--tserver_master_addrs=127.9.176.254:43673
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.193.keytab
--principal=kudu/127.9.176.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:36489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:22:09.029224 15359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:22:09.029484 15359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:22:09.029525 15359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:22:09.033540 15359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:22:09.033591 15359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:22:09.033661 15359 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.193
I20260808 22:22:09.038784 15359 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:36489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.193.keytab
--principal=kudu/127.9.176.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.176.193
--webserver_port=0
--tserver_master_addrs=127.9.176.254:43673
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15359
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.193
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:22:09.039773 15359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:22:09.040784 15359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:22:09.050149 15366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:09.050185 15367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:22:09.050237 15359 server_base.cc:1034] running on GCE node
W20260808 22:22:09.050185 15369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:22:09.050751 15359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:22:09.051342 15359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:22:09.052495 15359 hybrid_clock.cc:648] HybridClock initialized: now 1786227729052475 us; error 30 us; skew 500 ppm
Aug 08 22:22:09 dist-test-slave-ncp9 krb5kdc[15193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227729, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:22:09.055114 15359 init.cc:377] Logged in from keytab as kudu/127.9.176.193@KRBTEST.COM (short username kudu)
I20260808 22:22:09.055958 15359 webserver.cc:460] Webserver started at http://127.9.176.193:38083/ using document root <none> and password file <none>
I20260808 22:22:09.056468 15359 fs_manager.cc:359] Metadata directory not provided
I20260808 22:22:09.056533 15359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:22:09.056741 15359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:22:09.058728 15359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/minicluster-data/ts-0/data/instance:
uuid: "73a11ee6ee5148a7a9d07d0d1f28ebe1"
format_stamp: "Formatted at 2026-08-08 22:22:09 on dist-test-slave-ncp9"
server_key: "8ef65bc4330ac4e6b2783e26c4ba5240"
server_key_iv: "ad90eaa4d134d74d7137dd900e7e0a17"
server_key_version: "encryptionkey@0"
I20260808 22:22:09.059265 15359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/minicluster-data/ts-0/wal/instance:
uuid: "73a11ee6ee5148a7a9d07d0d1f28ebe1"
format_stamp: "Formatted at 2026-08-08 22:22:09 on dist-test-slave-ncp9"
server_key: "8ef65bc4330ac4e6b2783e26c4ba5240"
server_key_iv: "ad90eaa4d134d74d7137dd900e7e0a17"
server_key_version: "encryptionkey@0"
I20260808 22:22:09.062851 15359 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 22:22:09.065073 15384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:09.066179 15359 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 22:22:09.066282 15359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/minicluster-data/ts-0/wal
uuid: "73a11ee6ee5148a7a9d07d0d1f28ebe1"
format_stamp: "Formatted at 2026-08-08 22:22:09 on dist-test-slave-ncp9"
server_key: "8ef65bc4330ac4e6b2783e26c4ba5240"
server_key_iv: "ad90eaa4d134d74d7137dd900e7e0a17"
server_key_version: "encryptionkey@0"
I20260808 22:22:09.066371 15359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:22:09.083050 15359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:22:09.085621 15359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:22:09.085798 15359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:22:09.086341 15359 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:22:09.087118 15359 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:22:09.087167 15359 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:09.087218 15359 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:22:09.087247 15359 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:09.095572 15359 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.193:42603
I20260808 22:22:09.095642 15558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.193:42603 every 8 connection(s)
I20260808 22:22:09.096446 15359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
I20260808 22:22:09.103478 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 15359
I20260808 22:22:09.103595 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/minicluster-data/ts-0/wal/instance
I20260808 22:22:09.103873 9923 external_mini_cluster.cc:1428] Setting key a4dc71ee1920eecc9852140cee90786a
Aug 08 22:22:09 dist-test-slave-ncp9 krb5kdc[15193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227729, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.193@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
I20260808 22:22:09.109848 15322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:09.098143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.193:47783 (local address 127.9.176.254:43673)
0808 22:22:09.098301 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:09.098305 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:09.099174 (+ 869us) server_negotiation.cc:407] Connection header received
0808 22:22:09.100468 (+ 1294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:09.100471 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:09.100518 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:09.100589 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:09.101738 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:09.102287 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:09.103352 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:09.103574 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:09.103678 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:09.106370 (+ 2692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:09.106373 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:09.106375 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:22:09.106397 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:09.107918 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:09.108378 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:09.108382 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:09.108384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:09.108478 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:09.108789 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:09.108792 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:09.108794 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:09.109003 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:09.109079 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:09.109618 (+ 539us) server_negotiation.cc:299] Negotiation successful
0808 22:22:09.109725 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 22:22:09.110977 15562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:09.098569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:43673 (local address 127.9.176.193:47783)
0808 22:22:09.099026 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:09.099069 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:09.100285 (+ 1216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:09.100734 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:09.100741 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:09.101141 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:09.101619 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:09.101633 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:09.102374 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:09.102376 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:09.103251 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:09.103262 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:09.103722 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:09.103727 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:09.103890 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:09.104452 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:09.104468 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:09.106214 (+ 1746us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:22:09.108033 (+ 1819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:09.108038 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:09.108048 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:09.108287 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:09.108579 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:09.108581 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:09.108582 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:09.108706 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:09.109099 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:22:09.109105 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:22:09.109392 (+ 287us) client_negotiation.cc:769] Sending connection context
0808 22:22:09.109597 (+ 205us) client_negotiation.cc:240] Negotiation successful
0808 22:22:09.109909 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":116,"threads_started":1}
I20260808 22:22:09.112115 15560 heartbeater.cc:347] Connected to a master server at 127.9.176.254:43673
I20260808 22:22:09.112372 15560 heartbeater.cc:464] Registering TS with master...
I20260808 22:22:09.112972 15560 heartbeater.cc:511] Master 127.9.176.254:43673 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.9.176.194@KRBTEST.COM; defaulting to no policy
I20260808 22:22:09.114457 15264 ts_manager.cc:194] Registered new tserver with Master: 73a11ee6ee5148a7a9d07d0d1f28ebe1 (127.9.176.193:42603)
I20260808 22:22:09.115579 15264 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.176.193@KRBTEST.COM'} at 127.9.176.193:47783
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.176.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.176.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.194.keytab.
Entry for principal kudu/127.9.176.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.194.keytab.
I20260808 22:22:09.131381 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.194:0
--local_ip_for_outbound_sockets=127.9.176.194
--webserver_interface=127.9.176.194
--webserver_port=0
--tserver_master_addrs=127.9.176.254:43673
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.194.keytab
--principal=kudu/127.9.176.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:36489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:22:09.239650 15566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:22:09.239912 15566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:22:09.239982 15566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:22:09.244153 15566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:22:09.244220 15566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:22:09.244302 15566 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.194
I20260808 22:22:09.249716 15566 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:36489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.194.keytab
--principal=kudu/127.9.176.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.176.194
--webserver_port=0
--tserver_master_addrs=127.9.176.254:43673
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15566
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.194
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:22:09.250749 15566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:22:09.251773 15566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:22:09.261019 15574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:09.261053 15573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:09.261091 15577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:22:09.261245 15566 server_base.cc:1034] running on GCE node
I20260808 22:22:09.261659 15566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:22:09.262262 15566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:22:09.263406 15566 hybrid_clock.cc:648] HybridClock initialized: now 1786227729263383 us; error 37 us; skew 500 ppm
Aug 08 22:22:09 dist-test-slave-ncp9 krb5kdc[15193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227729, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:22:09.266000 15566 init.cc:377] Logged in from keytab as kudu/127.9.176.194@KRBTEST.COM (short username kudu)
I20260808 22:22:09.266846 15566 webserver.cc:460] Webserver started at http://127.9.176.194:43227/ using document root <none> and password file <none>
I20260808 22:22:09.267349 15566 fs_manager.cc:359] Metadata directory not provided
I20260808 22:22:09.267413 15566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:22:09.267617 15566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:22:09.269652 15566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/minicluster-data/ts-1/data/instance:
uuid: "7a919e4f25a149128b19436998a273f9"
format_stamp: "Formatted at 2026-08-08 22:22:09 on dist-test-slave-ncp9"
server_key: "b16d0ae2257d71064228e1b287dcf70f"
server_key_iv: "4da8af2e6b6d45ff5bb7bd2d5e78036e"
server_key_version: "encryptionkey@0"
I20260808 22:22:09.270206 15566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/minicluster-data/ts-1/wal/instance:
uuid: "7a919e4f25a149128b19436998a273f9"
format_stamp: "Formatted at 2026-08-08 22:22:09 on dist-test-slave-ncp9"
server_key: "b16d0ae2257d71064228e1b287dcf70f"
server_key_iv: "4da8af2e6b6d45ff5bb7bd2d5e78036e"
server_key_version: "encryptionkey@0"
I20260808 22:22:09.273695 15566 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 22:22:09.275874 15587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:09.277050 15566 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 22:22:09.277164 15566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/minicluster-data/ts-1/wal
uuid: "7a919e4f25a149128b19436998a273f9"
format_stamp: "Formatted at 2026-08-08 22:22:09 on dist-test-slave-ncp9"
server_key: "b16d0ae2257d71064228e1b287dcf70f"
server_key_iv: "4da8af2e6b6d45ff5bb7bd2d5e78036e"
server_key_version: "encryptionkey@0"
I20260808 22:22:09.277259 15566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:22:09.289662 15566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:22:09.292141 15566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:22:09.292300 15566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:22:09.292831 15566 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:22:09.293634 15566 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:22:09.293680 15566 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:09.293721 15566 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:22:09.293751 15566 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:09.302073 15566 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.194:33823
I20260808 22:22:09.302098 15747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.194:33823 every 8 connection(s)
I20260808 22:22:09.302904 15566 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
I20260808 22:22:09.306728 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 15566
I20260808 22:22:09.306835 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/minicluster-data/ts-1/wal/instance
I20260808 22:22:09.307137 9923 external_mini_cluster.cc:1428] Setting key 9b4720c80f575b2c6802cb98adf6dd25
Aug 08 22:22:09 dist-test-slave-ncp9 krb5kdc[15193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227729, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.194@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
I20260808 22:22:09.316589 15322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:09.304573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.194:48681 (local address 127.9.176.254:43673)
0808 22:22:09.304731 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:09.304736 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:09.305528 (+ 792us) server_negotiation.cc:407] Connection header received
0808 22:22:09.306875 (+ 1347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:09.306879 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:09.306939 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:09.307028 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:09.308335 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:09.309118 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:09.310327 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:09.310613 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:09.310694 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:09.313070 (+ 2376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:09.313074 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:09.313077 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:22:09.313105 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:09.314654 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:09.315121 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:09.315125 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:09.315127 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:09.315224 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:09.315504 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:09.315507 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:09.315509 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:09.315742 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:09.315842 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:09.316284 (+ 442us) server_negotiation.cc:299] Negotiation successful
0808 22:22:09.316411 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 22:22:09.317322 15752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:09.304954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:43673 (local address 127.9.176.194:48681)
0808 22:22:09.305381 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:09.305425 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:09.306661 (+ 1236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:09.307197 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:09.307206 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:09.307576 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:09.308156 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:09.308175 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:09.309271 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:09.309274 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:09.310192 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:09.310202 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:09.310739 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:09.310742 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:09.310893 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:09.311456 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:09.311472 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:09.312859 (+ 1387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:22:09.314761 (+ 1902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:09.314766 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:09.314782 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:09.315028 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:09.315304 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:09.315306 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:09.315308 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:09.315426 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:09.315830 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:22:09.315835 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:22:09.316059 (+ 224us) client_negotiation.cc:769] Sending connection context
0808 22:22:09.316209 (+ 150us) client_negotiation.cc:240] Negotiation successful
0808 22:22:09.316454 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":105,"threads_started":1}
WARNING: no policy specified for kudu/127.9.176.195@KRBTEST.COM; defaulting to no policy
I20260808 22:22:09.318403 15748 heartbeater.cc:347] Connected to a master server at 127.9.176.254:43673
I20260808 22:22:09.318634 15748 heartbeater.cc:464] Registering TS with master...
I20260808 22:22:09.319129 15748 heartbeater.cc:511] Master 127.9.176.254:43673 requested a full tablet report, sending...
I20260808 22:22:09.320138 15264 ts_manager.cc:194] Registered new tserver with Master: 7a919e4f25a149128b19436998a273f9 (127.9.176.194:33823)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.176.195@KRBTEST.COM" created.
I20260808 22:22:09.320868 15264 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.176.194@KRBTEST.COM'} at 127.9.176.194:48681
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.176.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.195.keytab.
Entry for principal kudu/127.9.176.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.195.keytab.
I20260808 22:22:09.333878 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.195:0
--local_ip_for_outbound_sockets=127.9.176.195
--webserver_interface=127.9.176.195
--webserver_port=0
--tserver_master_addrs=127.9.176.254:43673
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.195.keytab
--principal=kudu/127.9.176.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:36489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:22:09.447618 15756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:22:09.447916 15756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:22:09.447978 15756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:22:09.453245 15756 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:22:09.453300 15756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:22:09.453377 15756 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.195
I20260808 22:22:09.458583 15756 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:36489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.195.keytab
--principal=kudu/127.9.176.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.176.195
--webserver_port=0
--tserver_master_addrs=127.9.176.254:43673
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15756
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.195
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:22:09.459525 15756 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:22:09.460484 15756 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:22:09.469431 15761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:09.469520 15762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:09.469442 15765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:22:09.469694 15756 server_base.cc:1034] running on GCE node
I20260808 22:22:09.470304 15756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:22:09.471057 15756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:22:09.472241 15756 hybrid_clock.cc:648] HybridClock initialized: now 1786227729472215 us; error 38 us; skew 500 ppm
Aug 08 22:22:09 dist-test-slave-ncp9 krb5kdc[15193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227729, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:22:09.475586 15756 init.cc:377] Logged in from keytab as kudu/127.9.176.195@KRBTEST.COM (short username kudu)
I20260808 22:22:09.476682 15756 webserver.cc:460] Webserver started at http://127.9.176.195:44459/ using document root <none> and password file <none>
I20260808 22:22:09.477378 15756 fs_manager.cc:359] Metadata directory not provided
I20260808 22:22:09.477470 15756 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:22:09.477705 15756 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:22:09.480402 15756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/minicluster-data/ts-2/data/instance:
uuid: "dc9226c61d9744458685e11f01387564"
format_stamp: "Formatted at 2026-08-08 22:22:09 on dist-test-slave-ncp9"
server_key: "7c58365e31bfaa967bb33b53e23df5f5"
server_key_iv: "516959e5c34228c0a24680b3b7b72b0b"
server_key_version: "encryptionkey@0"
I20260808 22:22:09.481151 15756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/minicluster-data/ts-2/wal/instance:
uuid: "dc9226c61d9744458685e11f01387564"
format_stamp: "Formatted at 2026-08-08 22:22:09 on dist-test-slave-ncp9"
server_key: "7c58365e31bfaa967bb33b53e23df5f5"
server_key_iv: "516959e5c34228c0a24680b3b7b72b0b"
server_key_version: "encryptionkey@0"
I20260808 22:22:09.485652 15756 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 22:22:09.487928 15776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:09.489199 15756 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 22:22:09.489300 15756 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/minicluster-data/ts-2/wal
uuid: "dc9226c61d9744458685e11f01387564"
format_stamp: "Formatted at 2026-08-08 22:22:09 on dist-test-slave-ncp9"
server_key: "7c58365e31bfaa967bb33b53e23df5f5"
server_key_iv: "516959e5c34228c0a24680b3b7b72b0b"
server_key_version: "encryptionkey@0"
I20260808 22:22:09.489384 15756 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:22:09.512542 15756 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:22:09.515532 15756 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:22:09.515704 15756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:22:09.516260 15756 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:22:09.517072 15756 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:22:09.517127 15756 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:09.517181 15756 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:22:09.517210 15756 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:09.526173 15756 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.195:34965
I20260808 22:22:09.526213 15949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.195:34965 every 8 connection(s)
I20260808 22:22:09.527155 15756 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
I20260808 22:22:09.529634 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 15756
I20260808 22:22:09.529768 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786227706282457-9923-0/minicluster-data/ts-2/wal/instance
I20260808 22:22:09.530085 9923 external_mini_cluster.cc:1428] Setting key 56721c741b9580bc51991179c817dfdf
Aug 08 22:22:09 dist-test-slave-ncp9 krb5kdc[15193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227729, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.195@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
I20260808 22:22:09.543845 15322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:09.530767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.195:46937 (local address 127.9.176.254:43673)
0808 22:22:09.530879 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:09.530883 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:09.530898 (+ 15us) server_negotiation.cc:407] Connection header received
0808 22:22:09.532802 (+ 1904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:09.532806 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:09.532866 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:09.532979 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:09.534368 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:09.535228 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:09.536783 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:09.537205 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:09.537315 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:09.540101 (+ 2786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:09.540104 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:09.540106 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:22:09.540131 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:09.541428 (+ 1297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:09.542023 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:09.542026 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:09.542027 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:09.542098 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:09.542473 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:09.542476 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:09.542478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:09.542705 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:09.542792 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:09.543565 (+ 773us) server_negotiation.cc:299] Negotiation successful
0808 22:22:09.543688 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260808 22:22:09.544763 15954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:09.529668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:43673 (local address 127.9.176.195:46937)
0808 22:22:09.530213 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:09.530268 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:09.532558 (+ 2290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:09.533146 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:09.533154 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:09.533603 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:09.534213 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:09.534234 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:09.535363 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:09.535367 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:09.536645 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:09.536656 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:09.537402 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:09.537409 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:09.537577 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:09.538122 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:09.538139 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:09.539919 (+ 1780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:22:09.541547 (+ 1628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:09.541554 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:09.541573 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:09.541930 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:09.542190 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:09.542194 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:09.542196 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:09.542382 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:09.542969 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:22:09.542975 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:22:09.543294 (+ 319us) client_negotiation.cc:769] Sending connection context
0808 22:22:09.543503 (+ 209us) client_negotiation.cc:240] Negotiation successful
0808 22:22:09.543828 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":146,"threads_started":1}
I20260808 22:22:09.545996 15951 heartbeater.cc:347] Connected to a master server at 127.9.176.254:43673
I20260808 22:22:09.546247 15951 heartbeater.cc:464] Registering TS with master...
I20260808 22:22:09.546741 15951 heartbeater.cc:511] Master 127.9.176.254:43673 requested a full tablet report, sending...
I20260808 22:22:09.547861 15264 ts_manager.cc:194] Registered new tserver with Master: dc9226c61d9744458685e11f01387564 (127.9.176.195:34965)
I20260808 22:22:09.548640 15264 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.176.195@KRBTEST.COM'} at 127.9.176.195:46937
I20260808 22:22:09.556108 9923 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 22:22:09 dist-test-slave-ncp9 krb5kdc[15193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 22:22:09 dist-test-slave-ncp9 krb5kdc[15193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.176.193@KRBTEST.COM
I20260808 22:22:09.584107 15962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:09.574017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43434 (local address 127.9.176.193:42603)
0808 22:22:09.574415 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:09.574431 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:09.574460 (+ 29us) server_negotiation.cc:407] Connection header received
0808 22:22:09.574522 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:09.574528 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:09.574781 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:09.574948 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:09.575513 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:09.576778 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:09.577634 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:09.578241 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:09.578355 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:09.580881 (+ 2526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:09.580920 (+ 39us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:09.580937 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:22:09.580964 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:09.582421 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:09.582882 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:09.582886 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:09.582888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:09.582958 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:09.583207 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:09.583209 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:09.583210 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:09.583422 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:09.583587 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:09.583786 (+ 199us) server_negotiation.cc:299] Negotiation successful
0808 22:22:09.583907 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":89,"threads_started":1}
Aug 08 22:22:09 dist-test-slave-ncp9 krb5kdc[15193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
I20260808 22:22:09.594656 15322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:09.587147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35150 (local address 127.9.176.254:43673)
0808 22:22:09.587265 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:09.587269 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:09.587355 (+ 86us) server_negotiation.cc:407] Connection header received
0808 22:22:09.587476 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:09.587479 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:09.587527 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:09.587592 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:09.588135 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:09.588639 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:09.589406 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:09.589629 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:09.589715 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:09.591784 (+ 2069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:09.591788 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:09.591791 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:22:09.591817 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:09.593079 (+ 1262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:09.593433 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:09.593436 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:09.593438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:09.593521 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:09.593782 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:09.593786 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:09.593787 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:09.594000 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:09.594125 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:09.594364 (+ 239us) server_negotiation.cc:299] Negotiation successful
0808 22:22:09.594463 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260808 22:22:09.595009 15264 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35150
I20260808 22:22:09.595548 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 15359
I20260808 22:22:09.605628 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 15566
I20260808 22:22:09.624490 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 15756
I20260808 22:22:09.641427 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 15210
2026-08-08T22:22:09Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2247 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 22:22:09 dist-test-slave-ncp9 krb5kdc[15968](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 22:22:09 dist-test-slave-ncp9 krb5kdc[15968](info): set up 2 sockets
Aug 08 22:22:09 dist-test-slave-ncp9 krb5kdc[15968](info): commencing operation
krb5kdc: starting...
W20260808 22:22:10.754729 9923 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/krb5kdc/test-user.keytab.
Aug 08 22:22:10 dist-test-slave-ncp9 krb5kdc[15968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227730, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T22:22:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:22:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.176.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.176.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.176.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab.
Entry for principal kudu/127.9.176.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab.
I20260808 22:22:10.871882 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:44033
--webserver_interface=127.9.176.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab
--principal=kudu/127.9.176.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:40069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.176.254:44033
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:22:10.977442 15984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:22:10.977689 15984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:22:10.977744 15984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:22:10.981650 15984 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:22:10.981700 15984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:22:10.981725 15984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:22:10.981747 15984 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:22:10.981775 15984 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:22:10.987052 15984 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:40069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.176.254:44033
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab
--principal=kudu/127.9.176.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:44033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.176.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:22:10.988062 15984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:22:10.989120 15984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:22:10.997153 15992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:10.997156 15997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:22:10.997203 15984 server_base.cc:1034] running on GCE node
W20260808 22:22:10.997155 15994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:22:10.997905 15984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:22:10.998931 15984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:22:11.000095 15984 hybrid_clock.cc:648] HybridClock initialized: now 1786227731000078 us; error 28 us; skew 500 ppm
Aug 08 22:22:10 dist-test-slave-ncp9 krb5kdc[15968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227730, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:22:11.002683 15984 init.cc:377] Logged in from keytab as kudu/127.9.176.254@KRBTEST.COM (short username kudu)
I20260808 22:22:11.003378 15984 fs_manager.cc:359] Metadata directory not provided
I20260808 22:22:11.003448 15984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:22:11.003628 15984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:22:11.005646 15984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/minicluster-data/master-0/data/instance:
uuid: "9215d2f623834267805c7247fc03a62d"
format_stamp: "Formatted at 2026-08-08 22:22:10 on dist-test-slave-ncp9"
server_key: "d07b5d6fdde71dc42888c80ea1e0471e"
server_key_iv: "420726c0b85bcea3b37fbad8dae5acc2"
server_key_version: "encryptionkey@0"
I20260808 22:22:11.006188 15984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/minicluster-data/master-0/wal/instance:
uuid: "9215d2f623834267805c7247fc03a62d"
format_stamp: "Formatted at 2026-08-08 22:22:10 on dist-test-slave-ncp9"
server_key: "d07b5d6fdde71dc42888c80ea1e0471e"
server_key_iv: "420726c0b85bcea3b37fbad8dae5acc2"
server_key_version: "encryptionkey@0"
I20260808 22:22:11.009821 15984 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260808 22:22:11.012673 16006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:11.013880 15984 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 22:22:11.013994 15984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/minicluster-data/master-0/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/minicluster-data/master-0/wal
uuid: "9215d2f623834267805c7247fc03a62d"
format_stamp: "Formatted at 2026-08-08 22:22:10 on dist-test-slave-ncp9"
server_key: "d07b5d6fdde71dc42888c80ea1e0471e"
server_key_iv: "420726c0b85bcea3b37fbad8dae5acc2"
server_key_version: "encryptionkey@0"
I20260808 22:22:11.014093 15984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:22:11.034570 15984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:22:11.037115 15984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:22:11.037287 15984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:22:11.043744 15984 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.254:44033
I20260808 22:22:11.043970 16092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.254:44033 every 8 connection(s)
I20260808 22:22:11.044426 15984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
I20260808 22:22:11.047029 16093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:22:11.047178 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 15984
I20260808 22:22:11.047318 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/minicluster-data/master-0/wal/instance
I20260808 22:22:11.047652 9923 external_mini_cluster.cc:1428] Setting key fa517745f7cd37ee02a2e2248bca6d34
I20260808 22:22:11.053610 16093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9215d2f623834267805c7247fc03a62d: Bootstrap starting.
I20260808 22:22:11.056137 16093 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9215d2f623834267805c7247fc03a62d: Neither blocks nor log segments found. Creating new log.
Aug 08 22:22:11 dist-test-slave-ncp9 krb5kdc[15968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227730, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
I20260808 22:22:11.056843 16093 log.cc:824] T 00000000000000000000000000000000 P 9215d2f623834267805c7247fc03a62d: Log is configured to *not* fsync() on all Append() calls
I20260808 22:22:11.058697 16093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9215d2f623834267805c7247fc03a62d: No bootstrap required, opened a new log
I20260808 22:22:11.061208 16097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:11.048959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36930 (local address 127.9.176.254:44033)
0808 22:22:11.049348 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:11.049363 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:11.049397 (+ 34us) server_negotiation.cc:407] Connection header received
0808 22:22:11.050219 (+ 822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:11.050238 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:11.050546 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:11.050814 (+ 268us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:11.051672 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:11.052629 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:11.053792 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:11.054258 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:11.054377 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:11.056992 (+ 2615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:11.056999 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:11.057015 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:22:11.057044 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:11.058554 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:11.059011 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:11.059016 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:11.059021 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:11.059129 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:11.059387 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:11.059390 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:11.059391 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:11.059756 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:11.059943 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:11.060222 (+ 279us) server_negotiation.cc:299] Negotiation successful
0808 22:22:11.060476 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":114,"threads_started":1}
I20260808 22:22:11.062387 16093 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9215d2f623834267805c7247fc03a62d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9215d2f623834267805c7247fc03a62d" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 44033 } }
I20260808 22:22:11.062594 16093 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9215d2f623834267805c7247fc03a62d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:22:11.062650 16093 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9215d2f623834267805c7247fc03a62d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9215d2f623834267805c7247fc03a62d, State: Initialized, Role: FOLLOWER
I20260808 22:22:11.063041 16093 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9215d2f623834267805c7247fc03a62d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9215d2f623834267805c7247fc03a62d" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 44033 } }
I20260808 22:22:11.063197 16093 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9215d2f623834267805c7247fc03a62d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:22:11.063306 16093 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9215d2f623834267805c7247fc03a62d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:22:11.063406 16093 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9215d2f623834267805c7247fc03a62d [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:22:11.064193 16093 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9215d2f623834267805c7247fc03a62d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9215d2f623834267805c7247fc03a62d" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 44033 } }
I20260808 22:22:11.064495 16093 leader_election.cc:302] T 00000000000000000000000000000000 P 9215d2f623834267805c7247fc03a62d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9215d2f623834267805c7247fc03a62d; no voters:
I20260808 22:22:11.064752 16093 leader_election.cc:288] T 00000000000000000000000000000000 P 9215d2f623834267805c7247fc03a62d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:22:11.064862 16100 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9215d2f623834267805c7247fc03a62d [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:22:11.065091 16100 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9215d2f623834267805c7247fc03a62d [term 1 LEADER]: Becoming Leader. State: Replica: 9215d2f623834267805c7247fc03a62d, State: Running, Role: LEADER
I20260808 22:22:11.065394 16100 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9215d2f623834267805c7247fc03a62d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9215d2f623834267805c7247fc03a62d" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 44033 } }
I20260808 22:22:11.065832 16093 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9215d2f623834267805c7247fc03a62d [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:22:11.066883 16102 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9215d2f623834267805c7247fc03a62d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9215d2f623834267805c7247fc03a62d. Latest consensus state: current_term: 1 leader_uuid: "9215d2f623834267805c7247fc03a62d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9215d2f623834267805c7247fc03a62d" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 44033 } } }
I20260808 22:22:11.067039 16102 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9215d2f623834267805c7247fc03a62d [sys.catalog]: This master's current role is: LEADER
I20260808 22:22:11.067397 16114 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:22:11.067405 16101 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9215d2f623834267805c7247fc03a62d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9215d2f623834267805c7247fc03a62d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9215d2f623834267805c7247fc03a62d" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 44033 } } }
I20260808 22:22:11.068516 16101 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9215d2f623834267805c7247fc03a62d [sys.catalog]: This master's current role is: LEADER
I20260808 22:22:11.070403 16114 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:22:11.075388 16114 catalog_manager.cc:1347] Generated new cluster ID: 595a6c060dc24b8983e5e211e61cc0d3
I20260808 22:22:11.075501 16114 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:22:11.091539 16114 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:22:11.092427 16114 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:22:11.097369 16114 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9215d2f623834267805c7247fc03a62d: Generated new TSK 0
I20260808 22:22:11.098124 16114 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.176.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.176.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.176.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.193.keytab.
Entry for principal kudu/127.9.176.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.193.keytab.
I20260808 22:22:11.124346 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.193:0
--local_ip_for_outbound_sockets=127.9.176.193
--webserver_interface=127.9.176.193
--webserver_port=0
--tserver_master_addrs=127.9.176.254:44033
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.193.keytab
--principal=kudu/127.9.176.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:40069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:22:11.233191 16133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:22:11.233433 16133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:22:11.233475 16133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:22:11.237403 16133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:22:11.237458 16133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:22:11.237514 16133 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.193
I20260808 22:22:11.242663 16133 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:40069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.193.keytab
--principal=kudu/127.9.176.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.176.193
--webserver_port=0
--tserver_master_addrs=127.9.176.254:44033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16133
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.193
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:22:11.243616 16133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:22:11.244568 16133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:22:11.253614 16141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:11.253590 16140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:22:11.253741 16133 server_base.cc:1034] running on GCE node
W20260808 22:22:11.253602 16144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:22:11.254191 16133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:22:11.254760 16133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:22:11.255911 16133 hybrid_clock.cc:648] HybridClock initialized: now 1786227731255897 us; error 25 us; skew 500 ppm
Aug 08 22:22:11 dist-test-slave-ncp9 krb5kdc[15968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227731, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:22:11.258510 16133 init.cc:377] Logged in from keytab as kudu/127.9.176.193@KRBTEST.COM (short username kudu)
I20260808 22:22:11.259179 16133 fs_manager.cc:359] Metadata directory not provided
I20260808 22:22:11.259243 16133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:22:11.259413 16133 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:22:11.261387 16133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/minicluster-data/ts-0/data/instance:
uuid: "6c536abb11ac450d85cd1309f9a2fb29"
format_stamp: "Formatted at 2026-08-08 22:22:11 on dist-test-slave-ncp9"
server_key: "61a54536d66b32a32d46f01048bf032b"
server_key_iv: "0a64a9895559298149dfe3b14bddc641"
server_key_version: "encryptionkey@0"
I20260808 22:22:11.261886 16133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/minicluster-data/ts-0/wal/instance:
uuid: "6c536abb11ac450d85cd1309f9a2fb29"
format_stamp: "Formatted at 2026-08-08 22:22:11 on dist-test-slave-ncp9"
server_key: "61a54536d66b32a32d46f01048bf032b"
server_key_iv: "0a64a9895559298149dfe3b14bddc641"
server_key_version: "encryptionkey@0"
I20260808 22:22:11.265301 16133 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 22:22:11.267544 16154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:11.268661 16133 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 22:22:11.268770 16133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/minicluster-data/ts-0/wal
uuid: "6c536abb11ac450d85cd1309f9a2fb29"
format_stamp: "Formatted at 2026-08-08 22:22:11 on dist-test-slave-ncp9"
server_key: "61a54536d66b32a32d46f01048bf032b"
server_key_iv: "0a64a9895559298149dfe3b14bddc641"
server_key_version: "encryptionkey@0"
I20260808 22:22:11.268854 16133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:22:11.278242 16133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:22:11.280678 16133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:22:11.280835 16133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:22:11.281407 16133 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:22:11.282212 16133 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:22:11.282261 16133 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:11.282311 16133 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:22:11.282336 16133 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:11.290709 16133 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.193:43835
I20260808 22:22:11.290727 16324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.193:43835 every 8 connection(s)
I20260808 22:22:11.291230 16133 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
Aug 08 22:22:11 dist-test-slave-ncp9 krb5kdc[15968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227731, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.193@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
I20260808 22:22:11.300416 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 16133
I20260808 22:22:11.300529 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/minicluster-data/ts-0/wal/instance
I20260808 22:22:11.300854 9923 external_mini_cluster.cc:1428] Setting key 4b8f6f1cfc411889076cda3a62952901
I20260808 22:22:11.303802 16097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:11.292771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.193:53349 (local address 127.9.176.254:44033)
0808 22:22:11.292924 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:11.292929 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:11.293696 (+ 767us) server_negotiation.cc:407] Connection header received
0808 22:22:11.294891 (+ 1195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:11.294894 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:11.294951 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:11.295020 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:11.296107 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:11.296635 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:11.297789 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:11.298020 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:11.298110 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:11.300320 (+ 2210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:11.300324 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:11.300327 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:22:11.300355 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:11.301827 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:11.302405 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:11.302409 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:11.302411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:11.302505 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:11.302813 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:11.302816 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:11.302817 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:11.303031 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:11.303120 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:11.303536 (+ 416us) server_negotiation.cc:299] Negotiation successful
0808 22:22:11.303654 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 22:22:11.304479 16328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:11.293133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:44033 (local address 127.9.176.193:53349)
0808 22:22:11.293558 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:11.293598 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:11.294692 (+ 1094us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:11.295197 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:11.295203 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:11.295526 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:11.295964 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:11.295977 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:11.296759 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:11.296762 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:11.297669 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:11.297679 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:11.298163 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:11.298167 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:11.298320 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:11.298784 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:11.298799 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:11.300145 (+ 1346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:22:11.301939 (+ 1794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:11.301945 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:11.301958 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:11.302307 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:11.302588 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:11.302590 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:11.302592 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:11.302728 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:11.303119 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:22:11.303124 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:22:11.303333 (+ 209us) client_negotiation.cc:769] Sending connection context
0808 22:22:11.303474 (+ 141us) client_negotiation.cc:240] Negotiation successful
0808 22:22:11.303713 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":108,"threads_started":1}
I20260808 22:22:11.305580 16326 heartbeater.cc:347] Connected to a master server at 127.9.176.254:44033
I20260808 22:22:11.305817 16326 heartbeater.cc:464] Registering TS with master...
I20260808 22:22:11.306263 16326 heartbeater.cc:511] Master 127.9.176.254:44033 requested a full tablet report, sending...
I20260808 22:22:11.307695 16035 ts_manager.cc:194] Registered new tserver with Master: 6c536abb11ac450d85cd1309f9a2fb29 (127.9.176.193:43835)
I20260808 22:22:11.308782 16035 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.176.193@KRBTEST.COM'} at 127.9.176.193:53349
WARNING: no policy specified for kudu/127.9.176.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.176.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.176.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.194.keytab.
Entry for principal kudu/127.9.176.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.194.keytab.
I20260808 22:22:11.325788 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.194:0
--local_ip_for_outbound_sockets=127.9.176.194
--webserver_interface=127.9.176.194
--webserver_port=0
--tserver_master_addrs=127.9.176.254:44033
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.194.keytab
--principal=kudu/127.9.176.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:40069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:22:11.433350 16332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:22:11.433581 16332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:22:11.433626 16332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:22:11.437502 16332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:22:11.437551 16332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:22:11.437621 16332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.194
I20260808 22:22:11.442728 16332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:40069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.194.keytab
--principal=kudu/127.9.176.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.176.194
--webserver_port=0
--tserver_master_addrs=127.9.176.254:44033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16332
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.194
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:22:11.443935 16332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:22:11.445252 16332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:22:11.455040 16347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:11.455049 16343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:11.455044 16344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:22:11.455077 16332 server_base.cc:1034] running on GCE node
I20260808 22:22:11.455704 16332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:22:11.456283 16332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:22:11.457424 16332 hybrid_clock.cc:648] HybridClock initialized: now 1786227731457400 us; error 38 us; skew 500 ppm
Aug 08 22:22:11 dist-test-slave-ncp9 krb5kdc[15968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227731, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:22:11.459973 16332 init.cc:377] Logged in from keytab as kudu/127.9.176.194@KRBTEST.COM (short username kudu)
I20260808 22:22:11.460681 16332 fs_manager.cc:359] Metadata directory not provided
I20260808 22:22:11.460749 16332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:22:11.460968 16332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:22:11.462944 16332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/minicluster-data/ts-1/data/instance:
uuid: "689205a374dc4814a2e97cb1448884da"
format_stamp: "Formatted at 2026-08-08 22:22:11 on dist-test-slave-ncp9"
server_key: "95832f2b60c1ba8b986da5c9838b6d1d"
server_key_iv: "76b42980a430988e555394bb54021796"
server_key_version: "encryptionkey@0"
I20260808 22:22:11.463490 16332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/minicluster-data/ts-1/wal/instance:
uuid: "689205a374dc4814a2e97cb1448884da"
format_stamp: "Formatted at 2026-08-08 22:22:11 on dist-test-slave-ncp9"
server_key: "95832f2b60c1ba8b986da5c9838b6d1d"
server_key_iv: "76b42980a430988e555394bb54021796"
server_key_version: "encryptionkey@0"
I20260808 22:22:11.467128 16332 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 22:22:11.469437 16353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:11.470624 16332 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 22:22:11.470726 16332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/minicluster-data/ts-1/wal
uuid: "689205a374dc4814a2e97cb1448884da"
format_stamp: "Formatted at 2026-08-08 22:22:11 on dist-test-slave-ncp9"
server_key: "95832f2b60c1ba8b986da5c9838b6d1d"
server_key_iv: "76b42980a430988e555394bb54021796"
server_key_version: "encryptionkey@0"
I20260808 22:22:11.470813 16332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:22:11.485087 16332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:22:11.487708 16332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:22:11.487869 16332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:22:11.488397 16332 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:22:11.489204 16332 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:22:11.489259 16332 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:11.489308 16332 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:22:11.489331 16332 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:11.497459 16332 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.194:34757
I20260808 22:22:11.497479 16521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.194:34757 every 8 connection(s)
I20260808 22:22:11.497990 16332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
I20260808 22:22:11.501003 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 16332
I20260808 22:22:11.501116 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/minicluster-data/ts-1/wal/instance
I20260808 22:22:11.501395 9923 external_mini_cluster.cc:1428] Setting key bfa905014aeb90a1b2478fe3a9a14737
Aug 08 22:22:11 dist-test-slave-ncp9 krb5kdc[15968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227731, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.194@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
I20260808 22:22:11.512091 16097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:11.499649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.194:56069 (local address 127.9.176.254:44033)
0808 22:22:11.499783 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:11.499787 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:11.500592 (+ 805us) server_negotiation.cc:407] Connection header received
0808 22:22:11.501889 (+ 1297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:11.501893 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:11.501944 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:11.502015 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:11.503515 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:11.504230 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:11.505655 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:11.505877 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:11.505956 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:11.508568 (+ 2612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:11.508573 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:11.508576 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:22:11.508605 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:11.510109 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:11.510577 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:11.510581 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:11.510583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:11.510680 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:11.510974 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:11.510977 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:11.510979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:11.511218 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:11.511330 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:11.511770 (+ 440us) server_negotiation.cc:299] Negotiation successful
0808 22:22:11.511896 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
WARNING: no policy specified for kudu/127.9.176.195@KRBTEST.COM; defaulting to no policy
I20260808 22:22:11.512807 16525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:11.500003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:44033 (local address 127.9.176.194:56069)
0808 22:22:11.500441 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:11.500484 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:11.501682 (+ 1198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:11.502332 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:11.502340 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:11.502838 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:11.503363 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:11.503377 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:11.504335 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:11.504338 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:11.505538 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:11.505550 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:11.506023 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:11.506027 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:11.506217 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:11.506733 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:11.506753 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:11.508400 (+ 1647us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:22:11.510227 (+ 1827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:11.510231 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:11.510243 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:11.510480 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:11.510759 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:11.510761 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:11.510763 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:11.510889 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:11.511313 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:22:11.511318 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:22:11.511538 (+ 220us) client_negotiation.cc:769] Sending connection context
0808 22:22:11.511683 (+ 145us) client_negotiation.cc:240] Negotiation successful
0808 22:22:11.511921 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":110,"threads_started":1}
I20260808 22:22:11.513933 16523 heartbeater.cc:347] Connected to a master server at 127.9.176.254:44033
I20260808 22:22:11.514156 16523 heartbeater.cc:464] Registering TS with master...
I20260808 22:22:11.514647 16523 heartbeater.cc:511] Master 127.9.176.254:44033 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.176.195@KRBTEST.COM" created.
I20260808 22:22:11.515789 16035 ts_manager.cc:194] Registered new tserver with Master: 689205a374dc4814a2e97cb1448884da (127.9.176.194:34757)
I20260808 22:22:11.516505 16035 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.176.194@KRBTEST.COM'} at 127.9.176.194:56069
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.176.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.195.keytab.
Entry for principal kudu/127.9.176.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.195.keytab.
I20260808 22:22:11.529666 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.195:0
--local_ip_for_outbound_sockets=127.9.176.195
--webserver_interface=127.9.176.195
--webserver_port=0
--tserver_master_addrs=127.9.176.254:44033
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.195.keytab
--principal=kudu/127.9.176.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:40069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:22:11.637522 16529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:22:11.637770 16529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:22:11.637817 16529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:22:11.641737 16529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:22:11.641791 16529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:22:11.641861 16529 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.195
I20260808 22:22:11.646963 16529 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:40069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.195.keytab
--principal=kudu/127.9.176.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.176.195
--webserver_port=0
--tserver_master_addrs=127.9.176.254:44033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16529
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.195
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:22:11.647918 16529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:22:11.648947 16529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:22:11.657773 16537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:11.657768 16541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:22:11.657953 16529 server_base.cc:1034] running on GCE node
W20260808 22:22:11.657825 16538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:22:11.658452 16529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:22:11.659039 16529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:22:11.660198 16529 hybrid_clock.cc:648] HybridClock initialized: now 1786227731660173 us; error 36 us; skew 500 ppm
Aug 08 22:22:11 dist-test-slave-ncp9 krb5kdc[15968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227731, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:22:11.668215 16529 init.cc:377] Logged in from keytab as kudu/127.9.176.195@KRBTEST.COM (short username kudu)
I20260808 22:22:11.668948 16529 fs_manager.cc:359] Metadata directory not provided
I20260808 22:22:11.669013 16529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:22:11.669225 16529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:22:11.671258 16529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/minicluster-data/ts-2/data/instance:
uuid: "f4030245d1694fcd8f3a9c809f0f7c10"
format_stamp: "Formatted at 2026-08-08 22:22:11 on dist-test-slave-ncp9"
server_key: "0ebcfe3ce35c4e06c89fb525a5f22e1e"
server_key_iv: "716bbb61281a72eed305785ba296b0b6"
server_key_version: "encryptionkey@0"
I20260808 22:22:11.671849 16529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/minicluster-data/ts-2/wal/instance:
uuid: "f4030245d1694fcd8f3a9c809f0f7c10"
format_stamp: "Formatted at 2026-08-08 22:22:11 on dist-test-slave-ncp9"
server_key: "0ebcfe3ce35c4e06c89fb525a5f22e1e"
server_key_iv: "716bbb61281a72eed305785ba296b0b6"
server_key_version: "encryptionkey@0"
I20260808 22:22:11.675633 16529 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260808 22:22:11.677951 16551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:11.679076 16529 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 22:22:11.679188 16529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/minicluster-data/ts-2/wal
uuid: "f4030245d1694fcd8f3a9c809f0f7c10"
format_stamp: "Formatted at 2026-08-08 22:22:11 on dist-test-slave-ncp9"
server_key: "0ebcfe3ce35c4e06c89fb525a5f22e1e"
server_key_iv: "716bbb61281a72eed305785ba296b0b6"
server_key_version: "encryptionkey@0"
I20260808 22:22:11.679283 16529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:22:11.699224 16529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:22:11.702034 16529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:22:11.702237 16529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:22:11.702828 16529 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:22:11.703679 16529 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:22:11.703727 16529 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:11.703784 16529 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:22:11.703814 16529 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:11.712203 16529 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.195:46697
I20260808 22:22:11.712219 16726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.195:46697 every 8 connection(s)
I20260808 22:22:11.712809 16529 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
I20260808 22:22:11.715209 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 16529
I20260808 22:22:11.715329 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/minicluster-data/ts-2/wal/instance
I20260808 22:22:11.715642 9923 external_mini_cluster.cc:1428] Setting key 2496d416c976642ce2b59f0f8fd80434
Aug 08 22:22:11 dist-test-slave-ncp9 krb5kdc[15968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227731, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.195@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
I20260808 22:22:11.725874 16097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:11.714611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.195:46835 (local address 127.9.176.254:44033)
0808 22:22:11.714742 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:11.714747 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:11.715696 (+ 949us) server_negotiation.cc:407] Connection header received
0808 22:22:11.717161 (+ 1465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:11.717164 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:11.717221 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:11.717289 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:11.718522 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:11.719060 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:11.720202 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:11.720414 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:11.720497 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:11.722813 (+ 2316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:11.722816 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:11.722818 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:22:11.722841 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:11.723936 (+ 1095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:11.724400 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:11.724403 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:11.724404 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:11.724479 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:11.724794 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:11.724798 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:11.724800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:11.725011 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:11.725099 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:11.725579 (+ 480us) server_negotiation.cc:299] Negotiation successful
0808 22:22:11.725712 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 22:22:11.726635 16731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:11.715002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:44033 (local address 127.9.176.195:46835)
0808 22:22:11.715550 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:11.715593 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:11.716950 (+ 1357us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:11.717476 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:11.717483 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:11.717906 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:11.718386 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:11.718400 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:11.719177 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:11.719181 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:11.720077 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:11.720087 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:11.720549 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:11.720553 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:11.720738 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:11.721273 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:11.721289 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:11.722642 (+ 1353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:22:11.724042 (+ 1400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:11.724048 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:11.724063 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:11.724306 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:11.724561 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:11.724564 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:11.724566 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:11.724706 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:11.725108 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:22:11.725116 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:22:11.725354 (+ 238us) client_negotiation.cc:769] Sending connection context
0808 22:22:11.725520 (+ 166us) client_negotiation.cc:240] Negotiation successful
0808 22:22:11.725846 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":155,"threads_started":1}
I20260808 22:22:11.727645 16727 heartbeater.cc:347] Connected to a master server at 127.9.176.254:44033
I20260808 22:22:11.727898 16727 heartbeater.cc:464] Registering TS with master...
I20260808 22:22:11.728410 16727 heartbeater.cc:511] Master 127.9.176.254:44033 requested a full tablet report, sending...
I20260808 22:22:11.729468 16035 ts_manager.cc:194] Registered new tserver with Master: f4030245d1694fcd8f3a9c809f0f7c10 (127.9.176.195:46697)
I20260808 22:22:11.730018 16035 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.176.195@KRBTEST.COM'} at 127.9.176.195:46835
I20260808 22:22:11.740741 9923 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 22:22:11.752305 16097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:11.742866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36940 (local address 127.9.176.254:44033)
0808 22:22:11.742985 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:11.742989 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:11.743192 (+ 203us) server_negotiation.cc:407] Connection header received
0808 22:22:11.743314 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:11.743318 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:11.743360 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:11.743428 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:11.744101 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:11.744865 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:11.746036 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:11.746392 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:11.746486 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:11.747947 (+ 1461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:11.747950 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:11.747953 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:22:11.747978 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:11.750681 (+ 2703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:11.751183 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:11.751186 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:11.751188 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:11.751284 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:11.751573 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:11.751576 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:11.751578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:11.751824 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:11.751931 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:11.752090 (+ 159us) server_negotiation.cc:299] Negotiation successful
0808 22:22:11.752191 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260808 22:22:11.755016 16035 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36940:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 22:22:11.757972 16035 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 22:22:11.772152 16746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:11.765845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.195:46697 (local address 127.0.0.1:38934)
0808 22:22:11.767194 (+ 1349us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:11.767206 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:11.767291 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:11.767787 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:11.767789 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:11.767804 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:22:11.767879 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:11.767885 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:11.769213 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:11.769217 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:11.770441 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:11.770450 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:11.771567 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:11.771571 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:11.771714 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:11.771754 (+ 40us) client_negotiation.cc:769] Sending connection context
0808 22:22:11.771859 (+ 105us) client_negotiation.cc:240] Negotiation successful
0808 22:22:11.771971 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1252,"mutex_wait_us":41,"thread_start_us":95,"threads_started":1}
I20260808 22:22:11.772167 16749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:11.766488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38934 (local address 127.9.176.195:46697)
0808 22:22:11.766974 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:11.766986 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:11.767384 (+ 398us) server_negotiation.cc:407] Connection header received
0808 22:22:11.767464 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:11.767469 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:11.767651 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:11.767803 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:22:11.767972 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:11.769061 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:11.770561 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:11.771407 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:11.771547 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:11.771658 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:11.771912 (+ 254us) server_negotiation.cc:299] Negotiation successful
0808 22:22:11.772015 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":163,"threads_started":1}
I20260808 22:22:11.772620 16744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:11.765723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.194:34757 (local address 127.0.0.1:49678)
0808 22:22:11.766622 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:11.766687 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:11.766807 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:11.767434 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:11.767440 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:11.767461 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:22:11.767558 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:11.767563 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:11.768989 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:11.768992 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:11.770505 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:11.770517 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:11.771734 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:11.771737 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:11.771803 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:11.771821 (+ 18us) client_negotiation.cc:769] Sending connection context
0808 22:22:11.771887 (+ 66us) client_negotiation.cc:240] Negotiation successful
0808 22:22:11.772500 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":725,"mutex_wait_us":576,"thread_start_us":63,"threads_started":1}
I20260808 22:22:11.772226 16745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:11.765883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49678 (local address 127.9.176.194:34757)
0808 22:22:11.766366 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:11.766380 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:11.766952 (+ 572us) server_negotiation.cc:407] Connection header received
0808 22:22:11.767042 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:11.767047 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:11.767292 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:11.767443 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:22:11.767662 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:11.768832 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:11.770637 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:11.771407 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:11.771513 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:11.771658 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:11.771998 (+ 340us) server_negotiation.cc:299] Negotiation successful
0808 22:22:11.772082 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":133,"threads_started":1}
I20260808 22:22:11.773897 16750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:11.766699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50080 (local address 127.9.176.193:43835)
0808 22:22:11.767228 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:11.767239 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:11.769164 (+ 1925us) server_negotiation.cc:407] Connection header received
0808 22:22:11.769248 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:11.769254 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:11.769440 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:11.769578 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:22:11.769968 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:11.770914 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:11.772101 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:11.772918 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:11.773058 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:11.773207 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:11.773704 (+ 497us) server_negotiation.cc:299] Negotiation successful
0808 22:22:11.773788 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":112,"threads_started":1}
I20260808 22:22:11.774065 16743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:11.766516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.193:43835 (local address 127.0.0.1:50080)
0808 22:22:11.769051 (+ 2535us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:11.769063 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:11.769154 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:11.769749 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:11.769752 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:11.769768 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:22:11.769857 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:11.769864 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:11.771027 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:11.771030 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:11.771977 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:11.771987 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:11.773269 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:11.773272 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:11.773356 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:11.773379 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 22:22:11.773446 (+ 67us) client_negotiation.cc:240] Negotiation successful
0808 22:22:11.773510 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2476,"mutex_wait_us":4,"thread_start_us":71,"threads_started":1}
I20260808 22:22:11.775679 16623 tablet_service.cc:1467] Processing CreateTablet for tablet ab59db134bd6481aa1fac47532665c60 (DEFAULT_TABLE table=test-table [id=cd95da4f4bc34a3ebf72fa0dd640c9f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:22:11.775667 16425 tablet_service.cc:1467] Processing CreateTablet for tablet ab59db134bd6481aa1fac47532665c60 (DEFAULT_TABLE table=test-table [id=cd95da4f4bc34a3ebf72fa0dd640c9f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:22:11.776216 16228 tablet_service.cc:1467] Processing CreateTablet for tablet ab59db134bd6481aa1fac47532665c60 (DEFAULT_TABLE table=test-table [id=cd95da4f4bc34a3ebf72fa0dd640c9f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:22:11.776810 16623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab59db134bd6481aa1fac47532665c60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:22:11.776803 16425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab59db134bd6481aa1fac47532665c60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:22:11.777141 16228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab59db134bd6481aa1fac47532665c60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:22:11.782819 16752 tablet_bootstrap.cc:492] T ab59db134bd6481aa1fac47532665c60 P 6c536abb11ac450d85cd1309f9a2fb29: Bootstrap starting.
I20260808 22:22:11.784379 16754 tablet_bootstrap.cc:492] T ab59db134bd6481aa1fac47532665c60 P 689205a374dc4814a2e97cb1448884da: Bootstrap starting.
I20260808 22:22:11.784379 16753 tablet_bootstrap.cc:492] T ab59db134bd6481aa1fac47532665c60 P f4030245d1694fcd8f3a9c809f0f7c10: Bootstrap starting.
I20260808 22:22:11.785141 16752 tablet_bootstrap.cc:654] T ab59db134bd6481aa1fac47532665c60 P 6c536abb11ac450d85cd1309f9a2fb29: Neither blocks nor log segments found. Creating new log.
I20260808 22:22:11.786341 16752 log.cc:824] T ab59db134bd6481aa1fac47532665c60 P 6c536abb11ac450d85cd1309f9a2fb29: Log is configured to *not* fsync() on all Append() calls
I20260808 22:22:11.786486 16753 tablet_bootstrap.cc:654] T ab59db134bd6481aa1fac47532665c60 P f4030245d1694fcd8f3a9c809f0f7c10: Neither blocks nor log segments found. Creating new log.
I20260808 22:22:11.786895 16754 tablet_bootstrap.cc:654] T ab59db134bd6481aa1fac47532665c60 P 689205a374dc4814a2e97cb1448884da: Neither blocks nor log segments found. Creating new log.
I20260808 22:22:11.787714 16753 log.cc:824] T ab59db134bd6481aa1fac47532665c60 P f4030245d1694fcd8f3a9c809f0f7c10: Log is configured to *not* fsync() on all Append() calls
I20260808 22:22:11.787714 16754 log.cc:824] T ab59db134bd6481aa1fac47532665c60 P 689205a374dc4814a2e97cb1448884da: Log is configured to *not* fsync() on all Append() calls
I20260808 22:22:11.788025 16752 tablet_bootstrap.cc:492] T ab59db134bd6481aa1fac47532665c60 P 6c536abb11ac450d85cd1309f9a2fb29: No bootstrap required, opened a new log
I20260808 22:22:11.788285 16752 ts_tablet_manager.cc:1397] T ab59db134bd6481aa1fac47532665c60 P 6c536abb11ac450d85cd1309f9a2fb29: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260808 22:22:11.789497 16753 tablet_bootstrap.cc:492] T ab59db134bd6481aa1fac47532665c60 P f4030245d1694fcd8f3a9c809f0f7c10: No bootstrap required, opened a new log
I20260808 22:22:11.789728 16753 ts_tablet_manager.cc:1397] T ab59db134bd6481aa1fac47532665c60 P f4030245d1694fcd8f3a9c809f0f7c10: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260808 22:22:11.789915 16754 tablet_bootstrap.cc:492] T ab59db134bd6481aa1fac47532665c60 P 689205a374dc4814a2e97cb1448884da: No bootstrap required, opened a new log
I20260808 22:22:11.790099 16754 ts_tablet_manager.cc:1397] T ab59db134bd6481aa1fac47532665c60 P 689205a374dc4814a2e97cb1448884da: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 22:22:11.791971 16752 raft_consensus.cc:348] T ab59db134bd6481aa1fac47532665c60 P 6c536abb11ac450d85cd1309f9a2fb29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "689205a374dc4814a2e97cb1448884da" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34757 } } peers { permanent_uuid: "f4030245d1694fcd8f3a9c809f0f7c10" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46697 } } peers { permanent_uuid: "6c536abb11ac450d85cd1309f9a2fb29" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 43835 } }
I20260808 22:22:11.792240 16752 raft_consensus.cc:374] T ab59db134bd6481aa1fac47532665c60 P 6c536abb11ac450d85cd1309f9a2fb29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:22:11.792321 16752 raft_consensus.cc:729] T ab59db134bd6481aa1fac47532665c60 P 6c536abb11ac450d85cd1309f9a2fb29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c536abb11ac450d85cd1309f9a2fb29, State: Initialized, Role: FOLLOWER
I20260808 22:22:11.792426 16753 raft_consensus.cc:348] T ab59db134bd6481aa1fac47532665c60 P f4030245d1694fcd8f3a9c809f0f7c10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "689205a374dc4814a2e97cb1448884da" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34757 } } peers { permanent_uuid: "f4030245d1694fcd8f3a9c809f0f7c10" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46697 } } peers { permanent_uuid: "6c536abb11ac450d85cd1309f9a2fb29" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 43835 } }
I20260808 22:22:11.792680 16753 raft_consensus.cc:374] T ab59db134bd6481aa1fac47532665c60 P f4030245d1694fcd8f3a9c809f0f7c10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:22:11.792738 16753 raft_consensus.cc:729] T ab59db134bd6481aa1fac47532665c60 P f4030245d1694fcd8f3a9c809f0f7c10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4030245d1694fcd8f3a9c809f0f7c10, State: Initialized, Role: FOLLOWER
I20260808 22:22:11.792768 16752 consensus_queue.cc:260] T ab59db134bd6481aa1fac47532665c60 P 6c536abb11ac450d85cd1309f9a2fb29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "689205a374dc4814a2e97cb1448884da" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34757 } } peers { permanent_uuid: "f4030245d1694fcd8f3a9c809f0f7c10" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46697 } } peers { permanent_uuid: "6c536abb11ac450d85cd1309f9a2fb29" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 43835 } }
I20260808 22:22:11.793197 16753 consensus_queue.cc:260] T ab59db134bd6481aa1fac47532665c60 P f4030245d1694fcd8f3a9c809f0f7c10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "689205a374dc4814a2e97cb1448884da" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34757 } } peers { permanent_uuid: "f4030245d1694fcd8f3a9c809f0f7c10" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46697 } } peers { permanent_uuid: "6c536abb11ac450d85cd1309f9a2fb29" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 43835 } }
I20260808 22:22:11.793465 16326 heartbeater.cc:503] Master 127.9.176.254:44033 was elected leader, sending a full tablet report...
I20260808 22:22:11.793769 16752 ts_tablet_manager.cc:1428] T ab59db134bd6481aa1fac47532665c60 P 6c536abb11ac450d85cd1309f9a2fb29: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260808 22:22:11.794170 16753 ts_tablet_manager.cc:1428] T ab59db134bd6481aa1fac47532665c60 P f4030245d1694fcd8f3a9c809f0f7c10: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260808 22:22:11.794342 16727 heartbeater.cc:503] Master 127.9.176.254:44033 was elected leader, sending a full tablet report...
I20260808 22:22:11.794353 16754 raft_consensus.cc:348] T ab59db134bd6481aa1fac47532665c60 P 689205a374dc4814a2e97cb1448884da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "689205a374dc4814a2e97cb1448884da" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34757 } } peers { permanent_uuid: "f4030245d1694fcd8f3a9c809f0f7c10" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46697 } } peers { permanent_uuid: "6c536abb11ac450d85cd1309f9a2fb29" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 43835 } }
I20260808 22:22:11.794590 16754 raft_consensus.cc:374] T ab59db134bd6481aa1fac47532665c60 P 689205a374dc4814a2e97cb1448884da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:22:11.794637 16754 raft_consensus.cc:729] T ab59db134bd6481aa1fac47532665c60 P 689205a374dc4814a2e97cb1448884da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 689205a374dc4814a2e97cb1448884da, State: Initialized, Role: FOLLOWER
I20260808 22:22:11.795032 16754 consensus_queue.cc:260] T ab59db134bd6481aa1fac47532665c60 P 689205a374dc4814a2e97cb1448884da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "689205a374dc4814a2e97cb1448884da" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34757 } } peers { permanent_uuid: "f4030245d1694fcd8f3a9c809f0f7c10" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46697 } } peers { permanent_uuid: "6c536abb11ac450d85cd1309f9a2fb29" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 43835 } }
I20260808 22:22:11.795696 16523 heartbeater.cc:503] Master 127.9.176.254:44033 was elected leader, sending a full tablet report...
I20260808 22:22:11.795871 16754 ts_tablet_manager.cc:1428] T ab59db134bd6481aa1fac47532665c60 P 689205a374dc4814a2e97cb1448884da: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260808 22:22:11.908699 16762 raft_consensus.cc:482] T ab59db134bd6481aa1fac47532665c60 P f4030245d1694fcd8f3a9c809f0f7c10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 22:22:11.908968 16762 raft_consensus.cc:504] T ab59db134bd6481aa1fac47532665c60 P f4030245d1694fcd8f3a9c809f0f7c10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "689205a374dc4814a2e97cb1448884da" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34757 } } peers { permanent_uuid: "f4030245d1694fcd8f3a9c809f0f7c10" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46697 } } peers { permanent_uuid: "6c536abb11ac450d85cd1309f9a2fb29" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 43835 } }
I20260808 22:22:11.910130 16762 leader_election.cc:288] T ab59db134bd6481aa1fac47532665c60 P f4030245d1694fcd8f3a9c809f0f7c10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 689205a374dc4814a2e97cb1448884da (127.9.176.194:34757), 6c536abb11ac450d85cd1309f9a2fb29 (127.9.176.193:43835)
I20260808 22:22:11.913631 16731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:11.910387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.194:34757 (local address 127.9.176.195:55633)
0808 22:22:11.910531 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:11.910548 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:11.910654 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:11.910891 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:11.910893 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:11.910911 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:22:11.911036 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:11.911041 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:11.911761 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:11.911764 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:11.912570 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:11.912579 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:11.913244 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:11.913247 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:11.913348 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:11.913365 (+ 17us) client_negotiation.cc:769] Sending connection context
0808 22:22:11.913429 (+ 64us) client_negotiation.cc:240] Negotiation successful
0808 22:22:11.913497 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":20,"spinlock_wait_cycles":19712}
I20260808 22:22:11.913630 16745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:11.910504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.195:55633 (local address 127.9.176.194:34757)
0808 22:22:11.910626 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:11.910631 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:11.910645 (+ 14us) server_negotiation.cc:407] Connection header received
0808 22:22:11.910755 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:11.910758 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:11.910801 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:11.910861 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:22:11.911143 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:11.911645 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:11.912704 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:11.913132 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:11.913238 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:11.913291 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:11.913452 (+ 161us) server_negotiation.cc:299] Negotiation successful
0808 22:22:11.913519 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 22:22:11.913843 16750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:11.910507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.195:51801 (local address 127.9.176.193:43835)
0808 22:22:11.910634 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:11.910639 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:11.910932 (+ 293us) server_negotiation.cc:407] Connection header received
0808 22:22:11.911062 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:11.911065 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:11.911105 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:11.911165 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:22:11.911414 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:11.911889 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:11.912937 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:11.913338 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:11.913441 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:11.913491 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:11.913667 (+ 176us) server_negotiation.cc:299] Negotiation successful
0808 22:22:11.913735 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 22:22:11.913857 16764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:11.910387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.193:43835 (local address 127.9.176.195:51801)
0808 22:22:11.910829 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:11.910841 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:11.910960 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:11.911187 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:11.911190 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:11.911205 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:22:11.911309 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:11.911315 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:11.911995 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:11.911999 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:11.912785 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:11.912795 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:11.913447 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:11.913451 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:11.913555 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:11.913574 (+ 19us) client_negotiation.cc:769] Sending connection context
0808 22:22:11.913639 (+ 65us) client_negotiation.cc:240] Negotiation successful
0808 22:22:11.913715 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389,"mutex_wait_us":132,"thread_start_us":102,"threads_started":1}
I20260808 22:22:11.914415 16262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab59db134bd6481aa1fac47532665c60" candidate_uuid: "f4030245d1694fcd8f3a9c809f0f7c10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c536abb11ac450d85cd1309f9a2fb29" is_pre_election: true
I20260808 22:22:11.914414 16451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab59db134bd6481aa1fac47532665c60" candidate_uuid: "f4030245d1694fcd8f3a9c809f0f7c10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "689205a374dc4814a2e97cb1448884da" is_pre_election: true
I20260808 22:22:11.914700 16262 raft_consensus.cc:2454] T ab59db134bd6481aa1fac47532665c60 P 6c536abb11ac450d85cd1309f9a2fb29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4030245d1694fcd8f3a9c809f0f7c10 in term 0.
I20260808 22:22:11.914701 16451 raft_consensus.cc:2454] T ab59db134bd6481aa1fac47532665c60 P 689205a374dc4814a2e97cb1448884da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4030245d1694fcd8f3a9c809f0f7c10 in term 0.
I20260808 22:22:11.915135 16558 leader_election.cc:302] T ab59db134bd6481aa1fac47532665c60 P f4030245d1694fcd8f3a9c809f0f7c10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c536abb11ac450d85cd1309f9a2fb29, f4030245d1694fcd8f3a9c809f0f7c10; no voters:
I20260808 22:22:11.915383 16762 raft_consensus.cc:2781] T ab59db134bd6481aa1fac47532665c60 P f4030245d1694fcd8f3a9c809f0f7c10 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 22:22:11.915458 16762 raft_consensus.cc:482] T ab59db134bd6481aa1fac47532665c60 P f4030245d1694fcd8f3a9c809f0f7c10 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 22:22:11.915488 16762 raft_consensus.cc:3037] T ab59db134bd6481aa1fac47532665c60 P f4030245d1694fcd8f3a9c809f0f7c10 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:22:11.916517 16762 raft_consensus.cc:504] T ab59db134bd6481aa1fac47532665c60 P f4030245d1694fcd8f3a9c809f0f7c10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "689205a374dc4814a2e97cb1448884da" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34757 } } peers { permanent_uuid: "f4030245d1694fcd8f3a9c809f0f7c10" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46697 } } peers { permanent_uuid: "6c536abb11ac450d85cd1309f9a2fb29" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 43835 } }
I20260808 22:22:11.916847 16762 leader_election.cc:288] T ab59db134bd6481aa1fac47532665c60 P f4030245d1694fcd8f3a9c809f0f7c10 [CANDIDATE]: Term 1 election: Requested vote from peers 689205a374dc4814a2e97cb1448884da (127.9.176.194:34757), 6c536abb11ac450d85cd1309f9a2fb29 (127.9.176.193:43835)
I20260808 22:22:11.917156 16451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab59db134bd6481aa1fac47532665c60" candidate_uuid: "f4030245d1694fcd8f3a9c809f0f7c10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "689205a374dc4814a2e97cb1448884da"
I20260808 22:22:11.917266 16451 raft_consensus.cc:3037] T ab59db134bd6481aa1fac47532665c60 P 689205a374dc4814a2e97cb1448884da [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:22:11.918290 16451 raft_consensus.cc:2454] T ab59db134bd6481aa1fac47532665c60 P 689205a374dc4814a2e97cb1448884da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4030245d1694fcd8f3a9c809f0f7c10 in term 1.
I20260808 22:22:11.918540 16262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab59db134bd6481aa1fac47532665c60" candidate_uuid: "f4030245d1694fcd8f3a9c809f0f7c10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c536abb11ac450d85cd1309f9a2fb29"
I20260808 22:22:11.918659 16262 raft_consensus.cc:3037] T ab59db134bd6481aa1fac47532665c60 P 6c536abb11ac450d85cd1309f9a2fb29 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:22:11.919589 16262 raft_consensus.cc:2454] T ab59db134bd6481aa1fac47532665c60 P 6c536abb11ac450d85cd1309f9a2fb29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4030245d1694fcd8f3a9c809f0f7c10 in term 1.
I20260808 22:22:11.919879 16555 leader_election.cc:302] T ab59db134bd6481aa1fac47532665c60 P f4030245d1694fcd8f3a9c809f0f7c10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 689205a374dc4814a2e97cb1448884da, 6c536abb11ac450d85cd1309f9a2fb29, f4030245d1694fcd8f3a9c809f0f7c10; no voters:
I20260808 22:22:11.920032 16762 raft_consensus.cc:2781] T ab59db134bd6481aa1fac47532665c60 P f4030245d1694fcd8f3a9c809f0f7c10 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:22:11.920253 16762 raft_consensus.cc:686] T ab59db134bd6481aa1fac47532665c60 P f4030245d1694fcd8f3a9c809f0f7c10 [term 1 LEADER]: Becoming Leader. State: Replica: f4030245d1694fcd8f3a9c809f0f7c10, State: Running, Role: LEADER
I20260808 22:22:11.920572 16762 consensus_queue.cc:237] T ab59db134bd6481aa1fac47532665c60 P f4030245d1694fcd8f3a9c809f0f7c10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "689205a374dc4814a2e97cb1448884da" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34757 } } peers { permanent_uuid: "f4030245d1694fcd8f3a9c809f0f7c10" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46697 } } peers { permanent_uuid: "6c536abb11ac450d85cd1309f9a2fb29" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 43835 } }
I20260808 22:22:11.923734 16035 catalog_manager.cc:5495] T ab59db134bd6481aa1fac47532665c60 P f4030245d1694fcd8f3a9c809f0f7c10 reported cstate change: term changed from 0 to 1, leader changed from <none> to f4030245d1694fcd8f3a9c809f0f7c10 (127.9.176.195). New cstate: current_term: 1 leader_uuid: "f4030245d1694fcd8f3a9c809f0f7c10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "689205a374dc4814a2e97cb1448884da" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34757 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4030245d1694fcd8f3a9c809f0f7c10" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 46697 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c536abb11ac450d85cd1309f9a2fb29" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 43835 } health_report { overall_health: UNKNOWN } } }
I20260808 22:22:11.961658 16749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:11.958581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38936 (local address 127.9.176.195:46697)
0808 22:22:11.958712 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:11.958716 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:11.958732 (+ 16us) server_negotiation.cc:407] Connection header received
0808 22:22:11.958811 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:11.958813 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:11.958869 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:11.958980 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:22:11.959167 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:11.959675 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:11.960436 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:11.960641 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:11.960712 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:11.960962 (+ 250us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:22:11.961366 (+ 404us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:22:11.961466 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:11.961494 (+ 28us) server_negotiation.cc:299] Negotiation successful
0808 22:22:11.961544 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260808 22:22:11.963642 16728 tablet.cc:2369] T ab59db134bd6481aa1fac47532665c60 P f4030245d1694fcd8f3a9c809f0f7c10: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 22:22:11.967478 16451 raft_consensus.cc:1261] T ab59db134bd6481aa1fac47532665c60 P 689205a374dc4814a2e97cb1448884da [term 1 FOLLOWER]: Refusing update from remote peer f4030245d1694fcd8f3a9c809f0f7c10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:22:11.968125 16262 raft_consensus.cc:1261] T ab59db134bd6481aa1fac47532665c60 P 6c536abb11ac450d85cd1309f9a2fb29 [term 1 FOLLOWER]: Refusing update from remote peer f4030245d1694fcd8f3a9c809f0f7c10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:22:11.968164 16762 consensus_queue.cc:1035] T ab59db134bd6481aa1fac47532665c60 P f4030245d1694fcd8f3a9c809f0f7c10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "689205a374dc4814a2e97cb1448884da" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 34757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:22:11.968748 16762 consensus_queue.cc:1035] T ab59db134bd6481aa1fac47532665c60 P f4030245d1694fcd8f3a9c809f0f7c10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c536abb11ac450d85cd1309f9a2fb29" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 43835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:22:11.977819 16775 mvcc.cc:205] Tried to move back new op lower bound from 7316388790131625984 to 7316388789949325312. Current Snapshot: MvccSnapshot[applied={T|T < 7316388790131625984}]
I20260808 22:22:11.983160 16035 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36940:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:22:11.983309 16035 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36940:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:22:11.985554 16035 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9215d2f623834267805c7247fc03a62d: Sending DeleteTablet for 3 replicas of tablet ab59db134bd6481aa1fac47532665c60
I20260808 22:22:11.986202 16623 tablet_service.cc:1514] Processing DeleteTablet for tablet ab59db134bd6481aa1fac47532665c60 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:22:11 UTC) from {username='kudu', principal='kudu/127.9.176.254@KRBTEST.COM'} at 127.0.0.1:38934
I20260808 22:22:11.986228 16425 tablet_service.cc:1514] Processing DeleteTablet for tablet ab59db134bd6481aa1fac47532665c60 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:22:11 UTC) from {username='kudu', principal='kudu/127.9.176.254@KRBTEST.COM'} at 127.0.0.1:49678
I20260808 22:22:11.986639 16783 tablet_replica.cc:330] T ab59db134bd6481aa1fac47532665c60 P f4030245d1694fcd8f3a9c809f0f7c10: stopping tablet replica
I20260808 22:22:11.986667 16784 tablet_replica.cc:330] T ab59db134bd6481aa1fac47532665c60 P 689205a374dc4814a2e97cb1448884da: stopping tablet replica
I20260808 22:22:11.986936 16784 raft_consensus.cc:2229] T ab59db134bd6481aa1fac47532665c60 P 689205a374dc4814a2e97cb1448884da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:22:11.986936 16783 raft_consensus.cc:2229] T ab59db134bd6481aa1fac47532665c60 P f4030245d1694fcd8f3a9c809f0f7c10 [term 1 LEADER]: Raft consensus shutting down.
I20260808 22:22:11.987236 16783 raft_consensus.cc:2258] T ab59db134bd6481aa1fac47532665c60 P f4030245d1694fcd8f3a9c809f0f7c10 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:22:11.987236 16784 raft_consensus.cc:2258] T ab59db134bd6481aa1fac47532665c60 P 689205a374dc4814a2e97cb1448884da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:22:11.987205 16228 tablet_service.cc:1514] Processing DeleteTablet for tablet ab59db134bd6481aa1fac47532665c60 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:22:11 UTC) from {username='kudu', principal='kudu/127.9.176.254@KRBTEST.COM'} at 127.0.0.1:50080
I20260808 22:22:11.987879 16785 tablet_replica.cc:330] T ab59db134bd6481aa1fac47532665c60 P 6c536abb11ac450d85cd1309f9a2fb29: stopping tablet replica
I20260808 22:22:11.988201 16785 raft_consensus.cc:2229] T ab59db134bd6481aa1fac47532665c60 P 6c536abb11ac450d85cd1309f9a2fb29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:22:11.988256 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 16133
I20260808 22:22:11.988508 16785 raft_consensus.cc:2258] T ab59db134bd6481aa1fac47532665c60 P 6c536abb11ac450d85cd1309f9a2fb29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:22:11.988639 16783 ts_tablet_manager.cc:1905] T ab59db134bd6481aa1fac47532665c60 P f4030245d1694fcd8f3a9c809f0f7c10: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:22:11.991528 16783 ts_tablet_manager.cc:1918] T ab59db134bd6481aa1fac47532665c60 P f4030245d1694fcd8f3a9c809f0f7c10: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 22:22:11.991608 16783 log.cc:1196] T ab59db134bd6481aa1fac47532665c60 P f4030245d1694fcd8f3a9c809f0f7c10: Deleting WAL directory at /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/minicluster-data/ts-2/wal/wals/ab59db134bd6481aa1fac47532665c60
I20260808 22:22:11.991894 16783 ts_tablet_manager.cc:1939] T ab59db134bd6481aa1fac47532665c60 P f4030245d1694fcd8f3a9c809f0f7c10: Deleting consensus metadata
I20260808 22:22:11.992940 16009 catalog_manager.cc:4841] TS f4030245d1694fcd8f3a9c809f0f7c10 (127.9.176.195:46697): tablet ab59db134bd6481aa1fac47532665c60 (table test-table [id=cd95da4f4bc34a3ebf72fa0dd640c9f4]) successfully deleted
I20260808 22:22:11.996397 16784 ts_tablet_manager.cc:1905] T ab59db134bd6481aa1fac47532665c60 P 689205a374dc4814a2e97cb1448884da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:22:11.998996 16784 ts_tablet_manager.cc:1918] T ab59db134bd6481aa1fac47532665c60 P 689205a374dc4814a2e97cb1448884da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 22:22:11.999148 16784 log.cc:1196] T ab59db134bd6481aa1fac47532665c60 P 689205a374dc4814a2e97cb1448884da: Deleting WAL directory at /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786227706282457-9923-0/minicluster-data/ts-1/wal/wals/ab59db134bd6481aa1fac47532665c60
I20260808 22:22:11.999351 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 16332
W20260808 22:22:11.999370 16013 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.176.193:43835 (error 108)
I20260808 22:22:11.999475 16784 ts_tablet_manager.cc:1939] T ab59db134bd6481aa1fac47532665c60 P 689205a374dc4814a2e97cb1448884da: Deleting consensus metadata
I20260808 22:22:12.004467 16743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:12.004130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.193:43835 (local address 127.0.0.1:50082)
0808 22:22:12.004308 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:12.004380 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.193:43835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
W20260808 22:22:12.004681 16013 catalog_manager.cc:4584] TS 6c536abb11ac450d85cd1309f9a2fb29 (127.9.176.193:43835): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ab59db134bd6481aa1fac47532665c60: Network error: Client connection negotiation failed: client connection to 127.9.176.193:43835: connect: Connection refused (error 111)
I20260808 22:22:12.010171 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 16529
I20260808 22:22:12.019863 16743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:12.010258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.194:34757 (local address 127.0.0.1:49680)
0808 22:22:12.019702 (+ 9444us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:12.019771 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.194:34757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":9343}
I20260808 22:22:12.024415 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 15984
2026-08-08T22:22:12Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2383 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-08T22:22:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:22:12Z Disabled control of system clock
I20260808 22:22:12.059316 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786227706282457-9923-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786227706282457-9923-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786227706282457-9923-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:36427
--webserver_interface=127.9.176.254
--webserver_port=0
--builtin_ntp_servers=127.9.176.212:45811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.176.254:36427
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260808 22:22:12.166838 16793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:22:12.167068 16793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:22:12.167102 16793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:22:12.170907 16793 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:22:12.170961 16793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:22:12.170975 16793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:22:12.170989 16793 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:22:12.171003 16793 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:22:12.175680 16793 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:45811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786227706282457-9923-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786227706282457-9923-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.176.254:36427
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:36427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.176.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16793
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786227706282457-9923-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:22:12.176687 16793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:22:12.177997 16793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 22:22:12.186323 16793 server_base.cc:1034] running on GCE node
W20260808 22:22:12.186275 16799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:12.186280 16800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:12.186272 16802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:22:12.187065 16793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:22:12.188107 16793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:22:12.189263 16793 hybrid_clock.cc:648] HybridClock initialized: now 1786227732189242 us; error 32 us; skew 500 ppm
I20260808 22:22:12.190969 16793 webserver.cc:460] Webserver started at http://127.9.176.254:38383/ using document root <none> and password file <none>
I20260808 22:22:12.191493 16793 fs_manager.cc:359] Metadata directory not provided
I20260808 22:22:12.191560 16793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:22:12.191754 16793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:22:12.193755 16793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786227706282457-9923-0/minicluster-data/master-0/data/instance:
uuid: "35e9c6db6ceb49dda7d4ea0c1fb044cd"
format_stamp: "Formatted at 2026-08-08 22:22:12 on dist-test-slave-ncp9"
server_key: "379506452e09a001ae7c91d71f9d085d"
server_key_iv: "de224f11dce7a9f2ac4a7f7c1d50e55f"
server_key_version: "encryptionkey@0"
I20260808 22:22:12.194291 16793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786227706282457-9923-0/minicluster-data/master-0/wal/instance:
uuid: "35e9c6db6ceb49dda7d4ea0c1fb044cd"
format_stamp: "Formatted at 2026-08-08 22:22:12 on dist-test-slave-ncp9"
server_key: "379506452e09a001ae7c91d71f9d085d"
server_key_iv: "de224f11dce7a9f2ac4a7f7c1d50e55f"
server_key_version: "encryptionkey@0"
I20260808 22:22:12.197832 16793 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 22:22:12.200052 16813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:12.201236 16793 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 22:22:12.201345 16793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786227706282457-9923-0/minicluster-data/master-0/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786227706282457-9923-0/minicluster-data/master-0/wal
uuid: "35e9c6db6ceb49dda7d4ea0c1fb044cd"
format_stamp: "Formatted at 2026-08-08 22:22:12 on dist-test-slave-ncp9"
server_key: "379506452e09a001ae7c91d71f9d085d"
server_key_iv: "de224f11dce7a9f2ac4a7f7c1d50e55f"
server_key_version: "encryptionkey@0"
I20260808 22:22:12.201436 16793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786227706282457-9923-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786227706282457-9923-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786227706282457-9923-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:22:12.205771 16793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:22:12.206317 16793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:22:12.206473 16793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:22:12.212987 16793 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.254:36427
I20260808 22:22:12.212982 16895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.254:36427 every 8 connection(s)
I20260808 22:22:12.213881 16793 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
I20260808 22:22:12.214438 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 16793
I20260808 22:22:12.214550 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786227706282457-9923-0/minicluster-data/master-0/wal/instance
I20260808 22:22:12.214809 9923 external_mini_cluster.cc:1428] Setting key 1dbf2c6f04238a2b8456bbfd35b72277
I20260808 22:22:12.216993 16896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:22:12.220702 16901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:12.216117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54748 (local address 127.9.176.254:36427)
0808 22:22:12.217319 (+ 1202us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:12.217335 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:12.217369 (+ 34us) server_negotiation.cc:407] Connection header received
0808 22:22:12.218003 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:12.218019 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:12.218374 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:12.218685 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:12.219494 (+ 809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:12.219499 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:12.219511 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:22:12.219521 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:12.219578 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:22:12.219594 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:12.219721 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:12.219799 (+ 78us) server_negotiation.cc:299] Negotiation successful
0808 22:22:12.220036 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":837,"thread_start_us":688,"threads_started":1}
I20260808 22:22:12.222927 16896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35e9c6db6ceb49dda7d4ea0c1fb044cd: Bootstrap starting.
I20260808 22:22:12.225006 16896 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 35e9c6db6ceb49dda7d4ea0c1fb044cd: Neither blocks nor log segments found. Creating new log.
I20260808 22:22:12.225732 16896 log.cc:824] T 00000000000000000000000000000000 P 35e9c6db6ceb49dda7d4ea0c1fb044cd: Log is configured to *not* fsync() on all Append() calls
I20260808 22:22:12.227726 16896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35e9c6db6ceb49dda7d4ea0c1fb044cd: No bootstrap required, opened a new log
I20260808 22:22:12.231253 16896 raft_consensus.cc:348] T 00000000000000000000000000000000 P 35e9c6db6ceb49dda7d4ea0c1fb044cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35e9c6db6ceb49dda7d4ea0c1fb044cd" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 36427 } }
I20260808 22:22:12.231454 16896 raft_consensus.cc:374] T 00000000000000000000000000000000 P 35e9c6db6ceb49dda7d4ea0c1fb044cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:22:12.231506 16896 raft_consensus.cc:729] T 00000000000000000000000000000000 P 35e9c6db6ceb49dda7d4ea0c1fb044cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35e9c6db6ceb49dda7d4ea0c1fb044cd, State: Initialized, Role: FOLLOWER
I20260808 22:22:12.231882 16896 consensus_queue.cc:260] T 00000000000000000000000000000000 P 35e9c6db6ceb49dda7d4ea0c1fb044cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35e9c6db6ceb49dda7d4ea0c1fb044cd" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 36427 } }
I20260808 22:22:12.232002 16896 raft_consensus.cc:388] T 00000000000000000000000000000000 P 35e9c6db6ceb49dda7d4ea0c1fb044cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:22:12.232061 16896 raft_consensus.cc:482] T 00000000000000000000000000000000 P 35e9c6db6ceb49dda7d4ea0c1fb044cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:22:12.232165 16896 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 35e9c6db6ceb49dda7d4ea0c1fb044cd [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:22:12.233049 16896 raft_consensus.cc:504] T 00000000000000000000000000000000 P 35e9c6db6ceb49dda7d4ea0c1fb044cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35e9c6db6ceb49dda7d4ea0c1fb044cd" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 36427 } }
I20260808 22:22:12.233366 16896 leader_election.cc:302] T 00000000000000000000000000000000 P 35e9c6db6ceb49dda7d4ea0c1fb044cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 35e9c6db6ceb49dda7d4ea0c1fb044cd; no voters:
I20260808 22:22:12.233637 16896 leader_election.cc:288] T 00000000000000000000000000000000 P 35e9c6db6ceb49dda7d4ea0c1fb044cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:22:12.233749 16903 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 35e9c6db6ceb49dda7d4ea0c1fb044cd [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:22:12.233950 16903 raft_consensus.cc:686] T 00000000000000000000000000000000 P 35e9c6db6ceb49dda7d4ea0c1fb044cd [term 1 LEADER]: Becoming Leader. State: Replica: 35e9c6db6ceb49dda7d4ea0c1fb044cd, State: Running, Role: LEADER
I20260808 22:22:12.234301 16903 consensus_queue.cc:237] T 00000000000000000000000000000000 P 35e9c6db6ceb49dda7d4ea0c1fb044cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35e9c6db6ceb49dda7d4ea0c1fb044cd" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 36427 } }
I20260808 22:22:12.234763 16896 sys_catalog.cc:564] T 00000000000000000000000000000000 P 35e9c6db6ceb49dda7d4ea0c1fb044cd [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:22:12.235934 16905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35e9c6db6ceb49dda7d4ea0c1fb044cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 35e9c6db6ceb49dda7d4ea0c1fb044cd. Latest consensus state: current_term: 1 leader_uuid: "35e9c6db6ceb49dda7d4ea0c1fb044cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35e9c6db6ceb49dda7d4ea0c1fb044cd" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 36427 } } }
I20260808 22:22:12.236063 16905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35e9c6db6ceb49dda7d4ea0c1fb044cd [sys.catalog]: This master's current role is: LEADER
I20260808 22:22:12.236380 16904 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35e9c6db6ceb49dda7d4ea0c1fb044cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "35e9c6db6ceb49dda7d4ea0c1fb044cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35e9c6db6ceb49dda7d4ea0c1fb044cd" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 36427 } } }
I20260808 22:22:12.236464 16919 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:22:12.237035 16904 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35e9c6db6ceb49dda7d4ea0c1fb044cd [sys.catalog]: This master's current role is: LEADER
I20260808 22:22:12.239547 16919 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:22:12.244697 16919 catalog_manager.cc:1347] Generated new cluster ID: a691d35293524ec1bf455ea189a12270
I20260808 22:22:12.244755 16919 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:22:12.260408 16919 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:22:12.260694 16919 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:22:12.267529 16919 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 35e9c6db6ceb49dda7d4ea0c1fb044cd: Generated new TSK 0
I20260808 22:22:12.268108 16919 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 22:22:12.277809 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786227706282457-9923-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786227706282457-9923-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786227706282457-9923-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.193:0
--local_ip_for_outbound_sockets=127.9.176.193
--webserver_interface=127.9.176.193
--webserver_port=0
--tserver_master_addrs=127.9.176.254:36427
--builtin_ntp_servers=127.9.176.212:45811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260808 22:22:12.386253 16932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:22:12.386472 16932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:22:12.386507 16932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:22:12.390309 16932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:22:12.390350 16932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:22:12.390399 16932 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.193
I20260808 22:22:12.394876 16932 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:45811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786227706282457-9923-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786227706282457-9923-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.176.193
--webserver_port=0
--tserver_master_addrs=127.9.176.254:36427
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16932
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.193
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786227706282457-9923-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:22:12.395767 16932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:22:12.397028 16932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:22:12.406174 16940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:22:12.406212 16932 server_base.cc:1034] running on GCE node
W20260808 22:22:12.406176 16941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:12.406185 16943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:22:12.406745 16932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:22:12.407344 16932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:22:12.408499 16932 hybrid_clock.cc:648] HybridClock initialized: now 1786227732408485 us; error 29 us; skew 500 ppm
I20260808 22:22:12.410236 16932 webserver.cc:460] Webserver started at http://127.9.176.193:40869/ using document root <none> and password file <none>
I20260808 22:22:12.410768 16932 fs_manager.cc:359] Metadata directory not provided
I20260808 22:22:12.410836 16932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:22:12.411023 16932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:22:12.413007 16932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786227706282457-9923-0/minicluster-data/ts-0/data/instance:
uuid: "e91f1d98b12942f48116a93c065474d5"
format_stamp: "Formatted at 2026-08-08 22:22:12 on dist-test-slave-ncp9"
server_key: "05467827949ca191695cbf3fe6e321fe"
server_key_iv: "a49ed6d5c8ecbd7518772b2bca5ed4ce"
server_key_version: "encryptionkey@0"
I20260808 22:22:12.413533 16932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786227706282457-9923-0/minicluster-data/ts-0/wal/instance:
uuid: "e91f1d98b12942f48116a93c065474d5"
format_stamp: "Formatted at 2026-08-08 22:22:12 on dist-test-slave-ncp9"
server_key: "05467827949ca191695cbf3fe6e321fe"
server_key_iv: "a49ed6d5c8ecbd7518772b2bca5ed4ce"
server_key_version: "encryptionkey@0"
I20260808 22:22:12.416944 16932 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260808 22:22:12.419114 16952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:12.420202 16932 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 22:22:12.420321 16932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786227706282457-9923-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786227706282457-9923-0/minicluster-data/ts-0/wal
uuid: "e91f1d98b12942f48116a93c065474d5"
format_stamp: "Formatted at 2026-08-08 22:22:12 on dist-test-slave-ncp9"
server_key: "05467827949ca191695cbf3fe6e321fe"
server_key_iv: "a49ed6d5c8ecbd7518772b2bca5ed4ce"
server_key_version: "encryptionkey@0"
I20260808 22:22:12.420410 16932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786227706282457-9923-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786227706282457-9923-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786227706282457-9923-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:22:12.424643 16932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:22:12.425226 16932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:22:12.425379 16932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:22:12.425881 16932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:22:12.426718 16932 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:22:12.426766 16932 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:12.426815 16932 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:22:12.426842 16932 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:12.434653 16932 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.193:36921
I20260808 22:22:12.434705 17120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.193:36921 every 8 connection(s)
I20260808 22:22:12.435500 16932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
I20260808 22:22:12.441181 16901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:12.437368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.193:54463 (local address 127.9.176.254:36427)
0808 22:22:12.437507 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:12.437511 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:12.438488 (+ 977us) server_negotiation.cc:407] Connection header received
0808 22:22:12.439740 (+ 1252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:12.439743 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:12.439778 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:12.439845 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:12.440798 (+ 953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:12.440801 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:12.440803 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:22:12.440814 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:12.440833 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:22:12.440841 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:12.440931 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:12.441042 (+ 111us) server_negotiation.cc:299] Negotiation successful
0808 22:22:12.441076 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 22:22:12.441972 17123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:12.437735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:36427 (local address 127.9.176.193:54463)
0808 22:22:12.438350 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:12.438389 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:12.439560 (+ 1171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:12.439987 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:12.439994 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:12.440082 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:12.440563 (+ 481us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 22:22:12.440565 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:12.440602 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 22:22:12.440668 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:22:12.440918 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:22:12.440926 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:22:12.440931 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 22:22:12.441069 (+ 138us) client_negotiation.cc:240] Negotiation successful
0808 22:22:12.441271 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":106,"threads_started":1}
I20260808 22:22:12.443089 17121 heartbeater.cc:347] Connected to a master server at 127.9.176.254:36427
I20260808 22:22:12.443303 17121 heartbeater.cc:464] Registering TS with master...
I20260808 22:22:12.443663 17121 heartbeater.cc:511] Master 127.9.176.254:36427 requested a full tablet report, sending...
I20260808 22:22:12.444042 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 16932
I20260808 22:22:12.444149 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786227706282457-9923-0/minicluster-data/ts-0/wal/instance
I20260808 22:22:12.444469 9923 external_mini_cluster.cc:1428] Setting key 2f6c520dbeb68bbb43769515ccc90bd4
I20260808 22:22:12.445289 16839 ts_manager.cc:194] Registered new tserver with Master: e91f1d98b12942f48116a93c065474d5 (127.9.176.193:36921)
I20260808 22:22:12.446595 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786227706282457-9923-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786227706282457-9923-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786227706282457-9923-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.194:0
--local_ip_for_outbound_sockets=127.9.176.194
--webserver_interface=127.9.176.194
--webserver_port=0
--tserver_master_addrs=127.9.176.254:36427
--builtin_ntp_servers=127.9.176.212:45811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260808 22:22:12.553402 17128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:22:12.553637 17128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:22:12.553678 17128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:22:12.557641 17128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:22:12.557691 17128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:22:12.557758 17128 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.194
I20260808 22:22:12.562343 17128 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:45811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786227706282457-9923-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786227706282457-9923-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.176.194
--webserver_port=0
--tserver_master_addrs=127.9.176.254:36427
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17128
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.194
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786227706282457-9923-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:22:12.563328 17128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:22:12.564632 17128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:22:12.573669 17138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:12.573697 17137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:22:12.573813 17128 server_base.cc:1034] running on GCE node
W20260808 22:22:12.573968 17142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:22:12.574318 17128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:22:12.574877 17128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:22:12.576032 17128 hybrid_clock.cc:648] HybridClock initialized: now 1786227732576005 us; error 42 us; skew 500 ppm
I20260808 22:22:12.577760 17128 webserver.cc:460] Webserver started at http://127.9.176.194:40051/ using document root <none> and password file <none>
I20260808 22:22:12.578282 17128 fs_manager.cc:359] Metadata directory not provided
I20260808 22:22:12.578348 17128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:22:12.578526 17128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:22:12.580463 17128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786227706282457-9923-0/minicluster-data/ts-1/data/instance:
uuid: "64c9a129e120464aaca8d82402c8d7e2"
format_stamp: "Formatted at 2026-08-08 22:22:12 on dist-test-slave-ncp9"
server_key: "8ac62d11debdfdfe17f536c39a560959"
server_key_iv: "6cbef34045d52d9632dec5274a68201a"
server_key_version: "encryptionkey@0"
I20260808 22:22:12.581009 17128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786227706282457-9923-0/minicluster-data/ts-1/wal/instance:
uuid: "64c9a129e120464aaca8d82402c8d7e2"
format_stamp: "Formatted at 2026-08-08 22:22:12 on dist-test-slave-ncp9"
server_key: "8ac62d11debdfdfe17f536c39a560959"
server_key_iv: "6cbef34045d52d9632dec5274a68201a"
server_key_version: "encryptionkey@0"
I20260808 22:22:12.584400 17128 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260808 22:22:12.586630 17155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:12.587719 17128 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 22:22:12.587833 17128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786227706282457-9923-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786227706282457-9923-0/minicluster-data/ts-1/wal
uuid: "64c9a129e120464aaca8d82402c8d7e2"
format_stamp: "Formatted at 2026-08-08 22:22:12 on dist-test-slave-ncp9"
server_key: "8ac62d11debdfdfe17f536c39a560959"
server_key_iv: "6cbef34045d52d9632dec5274a68201a"
server_key_version: "encryptionkey@0"
I20260808 22:22:12.587922 17128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786227706282457-9923-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786227706282457-9923-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786227706282457-9923-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:22:12.592226 17128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:22:12.592764 17128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:22:12.592952 17128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:22:12.593465 17128 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:22:12.594310 17128 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:22:12.594357 17128 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:12.594405 17128 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:22:12.594434 17128 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:12.602552 17128 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.194:42967
I20260808 22:22:12.602588 17315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.194:42967 every 8 connection(s)
I20260808 22:22:12.603475 17128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
I20260808 22:22:12.609275 16901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:12.605182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.194:50995 (local address 127.9.176.254:36427)
0808 22:22:12.605310 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:12.605314 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:12.606340 (+ 1026us) server_negotiation.cc:407] Connection header received
0808 22:22:12.607702 (+ 1362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:12.607705 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:12.607750 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:12.607824 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:12.608822 (+ 998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:12.608826 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:12.608828 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:22:12.608841 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:12.608861 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:22:12.608869 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:12.608981 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:12.609103 (+ 122us) server_negotiation.cc:299] Negotiation successful
0808 22:22:12.609145 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 22:22:12.610073 17318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:12.605575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:36427 (local address 127.9.176.194:50995)
0808 22:22:12.606204 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:12.606243 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:12.607485 (+ 1242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:12.607994 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:12.608001 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:12.608084 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:12.608574 (+ 490us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 22:22:12.608575 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:12.608614 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 22:22:12.608682 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:22:12.608967 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:22:12.608975 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:22:12.608981 (+ 6us) client_negotiation.cc:769] Sending connection context
0808 22:22:12.609137 (+ 156us) client_negotiation.cc:240] Negotiation successful
0808 22:22:12.609398 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":128,"threads_started":1}
I20260808 22:22:12.611128 17316 heartbeater.cc:347] Connected to a master server at 127.9.176.254:36427
I20260808 22:22:12.611333 17316 heartbeater.cc:464] Registering TS with master...
I20260808 22:22:12.611703 17316 heartbeater.cc:511] Master 127.9.176.254:36427 requested a full tablet report, sending...
I20260808 22:22:12.612362 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 17128
I20260808 22:22:12.612459 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786227706282457-9923-0/minicluster-data/ts-1/wal/instance
I20260808 22:22:12.612649 16839 ts_manager.cc:194] Registered new tserver with Master: 64c9a129e120464aaca8d82402c8d7e2 (127.9.176.194:42967)
I20260808 22:22:12.612732 9923 external_mini_cluster.cc:1428] Setting key a0ec073bf497d7d43ddf1ce9b07c2373
I20260808 22:22:12.614549 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786227706282457-9923-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786227706282457-9923-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786227706282457-9923-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.195:0
--local_ip_for_outbound_sockets=127.9.176.195
--webserver_interface=127.9.176.195
--webserver_port=0
--tserver_master_addrs=127.9.176.254:36427
--builtin_ntp_servers=127.9.176.212:45811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260808 22:22:12.724355 17319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:22:12.724610 17319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:22:12.724650 17319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:22:12.728586 17319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:22:12.728642 17319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:22:12.728696 17319 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.195
I20260808 22:22:12.733371 17319 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:45811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786227706282457-9923-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786227706282457-9923-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.176.195
--webserver_port=0
--tserver_master_addrs=127.9.176.254:36427
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17319
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.195
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786227706282457-9923-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:22:12.734314 17319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:22:12.735747 17319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:22:12.744976 17328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:12.744964 17333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:22:12.745021 17319 server_base.cc:1034] running on GCE node
W20260808 22:22:12.744966 17335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:22:12.745604 17319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:22:12.746207 17319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:22:12.747380 17319 hybrid_clock.cc:648] HybridClock initialized: now 1786227732747351 us; error 39 us; skew 500 ppm
I20260808 22:22:12.749294 17319 webserver.cc:460] Webserver started at http://127.9.176.195:45805/ using document root <none> and password file <none>
I20260808 22:22:12.749846 17319 fs_manager.cc:359] Metadata directory not provided
I20260808 22:22:12.749914 17319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:22:12.750125 17319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:22:12.752270 17319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786227706282457-9923-0/minicluster-data/ts-2/data/instance:
uuid: "675f6a806406467fadd299d497d5ef1b"
format_stamp: "Formatted at 2026-08-08 22:22:12 on dist-test-slave-ncp9"
server_key: "c59963f095db503ca0599334cf8d2ace"
server_key_iv: "23c20f034f36080202fc1f7a5f27b5de"
server_key_version: "encryptionkey@0"
I20260808 22:22:12.752832 17319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786227706282457-9923-0/minicluster-data/ts-2/wal/instance:
uuid: "675f6a806406467fadd299d497d5ef1b"
format_stamp: "Formatted at 2026-08-08 22:22:12 on dist-test-slave-ncp9"
server_key: "c59963f095db503ca0599334cf8d2ace"
server_key_iv: "23c20f034f36080202fc1f7a5f27b5de"
server_key_version: "encryptionkey@0"
I20260808 22:22:12.756421 17319 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260808 22:22:12.758787 17345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:12.759910 17319 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 22:22:12.760040 17319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786227706282457-9923-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786227706282457-9923-0/minicluster-data/ts-2/wal
uuid: "675f6a806406467fadd299d497d5ef1b"
format_stamp: "Formatted at 2026-08-08 22:22:12 on dist-test-slave-ncp9"
server_key: "c59963f095db503ca0599334cf8d2ace"
server_key_iv: "23c20f034f36080202fc1f7a5f27b5de"
server_key_version: "encryptionkey@0"
I20260808 22:22:12.760136 17319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786227706282457-9923-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786227706282457-9923-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786227706282457-9923-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:22:12.764621 17319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:22:12.765239 17319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:22:12.765406 17319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:22:12.765992 17319 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:22:12.766897 17319 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:22:12.766952 17319 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:12.767005 17319 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:22:12.767037 17319 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:12.775934 17319 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.195:36915
I20260808 22:22:12.775955 17508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.195:36915 every 8 connection(s)
I20260808 22:22:12.776839 17319 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
I20260808 22:22:12.781070 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 17319
I20260808 22:22:12.781173 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786227706282457-9923-0/minicluster-data/ts-2/wal/instance
I20260808 22:22:12.781410 9923 external_mini_cluster.cc:1428] Setting key efb349dabff17a168a73b91ee5a700e4
I20260808 22:22:12.782507 16901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:12.778519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.195:50771 (local address 127.9.176.254:36427)
0808 22:22:12.778643 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:12.778647 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:12.779616 (+ 969us) server_negotiation.cc:407] Connection header received
0808 22:22:12.780866 (+ 1250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:12.780870 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:12.780934 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:12.781013 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:12.782026 (+ 1013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:12.782029 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:12.782032 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:22:12.782044 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:12.782067 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:22:12.782076 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:12.782152 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:12.782335 (+ 183us) server_negotiation.cc:299] Negotiation successful
0808 22:22:12.782376 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260808 22:22:12.783383 17512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:12.778878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:36427 (local address 127.9.176.195:50771)
0808 22:22:12.779481 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:12.779517 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:12.780658 (+ 1141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:12.781186 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:12.781194 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:12.781298 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:12.781783 (+ 485us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 22:22:12.781785 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:12.781821 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 22:22:12.781886 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:22:12.782173 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:22:12.782191 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:22:12.782197 (+ 6us) client_negotiation.cc:769] Sending connection context
0808 22:22:12.782365 (+ 168us) client_negotiation.cc:240] Negotiation successful
0808 22:22:12.782593 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":111,"threads_started":1}
I20260808 22:22:12.784334 17509 heartbeater.cc:347] Connected to a master server at 127.9.176.254:36427
I20260808 22:22:12.784528 17509 heartbeater.cc:464] Registering TS with master...
I20260808 22:22:12.784880 17509 heartbeater.cc:511] Master 127.9.176.254:36427 requested a full tablet report, sending...
I20260808 22:22:12.785828 16839 ts_manager.cc:194] Registered new tserver with Master: 675f6a806406467fadd299d497d5ef1b (127.9.176.195:36915)
I20260808 22:22:12.795037 9923 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 22:22:12.798247 16901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:12.796785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54758 (local address 127.9.176.254:36427)
0808 22:22:12.796915 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:12.796919 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:12.797099 (+ 180us) server_negotiation.cc:407] Connection header received
0808 22:22:12.797222 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:12.797225 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:12.797255 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:12.797317 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:12.797916 (+ 599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:12.797920 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:12.797922 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:22:12.797933 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:12.797949 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:22:12.797956 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:12.798024 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:12.798097 (+ 73us) server_negotiation.cc:299] Negotiation successful
0808 22:22:12.798135 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260808 22:22:12.800570 16839 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54758:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 22:22:12.803474 16839 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 22:22:12.814548 17527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:12.811325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.195:36915 (local address 127.0.0.1:40402)
0808 22:22:12.812491 (+ 1166us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:12.812502 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:12.812577 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:12.813388 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:12.813392 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:12.813406 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:12.813826 (+ 420us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 22:22:12.813828 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:12.813840 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 22:22:12.813873 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:22:12.814192 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:22:12.814202 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:22:12.814208 (+ 6us) client_negotiation.cc:769] Sending connection context
0808 22:22:12.814321 (+ 113us) client_negotiation.cc:240] Negotiation successful
0808 22:22:12.814387 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1114,"thread_start_us":99,"threads_started":1}
I20260808 22:22:12.814596 17528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:12.811958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40402 (local address 127.9.176.195:36915)
0808 22:22:12.812361 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:12.812371 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:12.812607 (+ 236us) server_negotiation.cc:407] Connection header received
0808 22:22:12.812668 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:12.812674 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:12.812865 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:12.813031 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:12.814007 (+ 976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:12.814014 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:12.814027 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:22:12.814041 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:12.814084 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:22:12.814107 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:12.814212 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:12.814353 (+ 141us) server_negotiation.cc:299] Negotiation successful
0808 22:22:12.814429 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":122,"threads_started":1}
I20260808 22:22:12.814548 17524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:12.811229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.193:36921 (local address 127.0.0.1:54516)
0808 22:22:12.812044 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:12.812094 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:12.812181 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:12.813036 (+ 855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:12.813042 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:12.813122 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:12.813666 (+ 544us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 22:22:12.813668 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:12.813702 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 22:22:12.813770 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:22:12.814272 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:22:12.814275 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:22:12.814278 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 22:22:12.814333 (+ 55us) client_negotiation.cc:240] Negotiation successful
0808 22:22:12.814387 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"thread_start_us":61,"threads_started":1}
I20260808 22:22:12.815171 17526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:12.811348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54516 (local address 127.9.176.193:36921)
0808 22:22:12.812678 (+ 1330us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:12.812689 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:12.812708 (+ 19us) server_negotiation.cc:407] Connection header received
0808 22:22:12.812751 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:12.812756 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:12.812879 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:12.813031 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:12.814107 (+ 1076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:12.814113 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:12.814123 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:22:12.814133 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:12.814173 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:22:12.814200 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:12.814302 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:12.814897 (+ 595us) server_negotiation.cc:299] Negotiation successful
0808 22:22:12.814969 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1223,"thread_start_us":118,"threads_started":1}
I20260808 22:22:12.815474 17525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:12.811562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.194:42967 (local address 127.0.0.1:50914)
0808 22:22:12.812711 (+ 1149us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:12.812720 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:12.812780 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:12.813867 (+ 1087us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:12.813870 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:12.813884 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:12.814850 (+ 966us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 22:22:12.814852 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:12.814862 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 22:22:12.814889 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:22:12.815256 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:22:12.815259 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:22:12.815263 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 22:22:12.815330 (+ 67us) client_negotiation.cc:240] Negotiation successful
0808 22:22:12.815363 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1101,"thread_start_us":90,"threads_started":1}
I20260808 22:22:12.815900 17530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:12.811959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50914 (local address 127.9.176.194:42967)
0808 22:22:12.812475 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:12.812484 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:12.812805 (+ 321us) server_negotiation.cc:407] Connection header received
0808 22:22:12.812867 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:12.812872 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:12.813132 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:12.813303 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:12.814997 (+ 1694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:12.815003 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:12.815014 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:22:12.815027 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:12.815070 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:22:12.815105 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:12.815210 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:12.815683 (+ 473us) server_negotiation.cc:299] Negotiation successful
0808 22:22:12.815749 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":121,"threads_started":1}
I20260808 22:22:12.817880 17023 tablet_service.cc:1467] Processing CreateTablet for tablet b4e50a77129f45b19ac8084ba84f3c8f (DEFAULT_TABLE table=test-table [id=524d995258b74c9083bbb925e871bdab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:22:12.817934 17419 tablet_service.cc:1467] Processing CreateTablet for tablet b4e50a77129f45b19ac8084ba84f3c8f (DEFAULT_TABLE table=test-table [id=524d995258b74c9083bbb925e871bdab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:22:12.818297 17219 tablet_service.cc:1467] Processing CreateTablet for tablet b4e50a77129f45b19ac8084ba84f3c8f (DEFAULT_TABLE table=test-table [id=524d995258b74c9083bbb925e871bdab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:22:12.818770 17023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4e50a77129f45b19ac8084ba84f3c8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:22:12.818943 17419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4e50a77129f45b19ac8084ba84f3c8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:22:12.819185 17219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4e50a77129f45b19ac8084ba84f3c8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:22:12.824572 17531 tablet_bootstrap.cc:492] T b4e50a77129f45b19ac8084ba84f3c8f P e91f1d98b12942f48116a93c065474d5: Bootstrap starting.
I20260808 22:22:12.826231 17538 tablet_bootstrap.cc:492] T b4e50a77129f45b19ac8084ba84f3c8f P 675f6a806406467fadd299d497d5ef1b: Bootstrap starting.
I20260808 22:22:12.826594 17532 tablet_bootstrap.cc:492] T b4e50a77129f45b19ac8084ba84f3c8f P 64c9a129e120464aaca8d82402c8d7e2: Bootstrap starting.
I20260808 22:22:12.826786 17531 tablet_bootstrap.cc:654] T b4e50a77129f45b19ac8084ba84f3c8f P e91f1d98b12942f48116a93c065474d5: Neither blocks nor log segments found. Creating new log.
I20260808 22:22:12.827622 17531 log.cc:824] T b4e50a77129f45b19ac8084ba84f3c8f P e91f1d98b12942f48116a93c065474d5: Log is configured to *not* fsync() on all Append() calls
I20260808 22:22:12.828548 17532 tablet_bootstrap.cc:654] T b4e50a77129f45b19ac8084ba84f3c8f P 64c9a129e120464aaca8d82402c8d7e2: Neither blocks nor log segments found. Creating new log.
I20260808 22:22:12.829252 17532 log.cc:824] T b4e50a77129f45b19ac8084ba84f3c8f P 64c9a129e120464aaca8d82402c8d7e2: Log is configured to *not* fsync() on all Append() calls
I20260808 22:22:12.829756 17531 tablet_bootstrap.cc:492] T b4e50a77129f45b19ac8084ba84f3c8f P e91f1d98b12942f48116a93c065474d5: No bootstrap required, opened a new log
I20260808 22:22:12.829952 17538 tablet_bootstrap.cc:654] T b4e50a77129f45b19ac8084ba84f3c8f P 675f6a806406467fadd299d497d5ef1b: Neither blocks nor log segments found. Creating new log.
I20260808 22:22:12.829972 17531 ts_tablet_manager.cc:1397] T b4e50a77129f45b19ac8084ba84f3c8f P e91f1d98b12942f48116a93c065474d5: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 22:22:12.830724 17538 log.cc:824] T b4e50a77129f45b19ac8084ba84f3c8f P 675f6a806406467fadd299d497d5ef1b: Log is configured to *not* fsync() on all Append() calls
I20260808 22:22:12.830839 17532 tablet_bootstrap.cc:492] T b4e50a77129f45b19ac8084ba84f3c8f P 64c9a129e120464aaca8d82402c8d7e2: No bootstrap required, opened a new log
I20260808 22:22:12.831008 17532 ts_tablet_manager.cc:1397] T b4e50a77129f45b19ac8084ba84f3c8f P 64c9a129e120464aaca8d82402c8d7e2: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260808 22:22:12.833582 17531 raft_consensus.cc:348] T b4e50a77129f45b19ac8084ba84f3c8f P e91f1d98b12942f48116a93c065474d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91f1d98b12942f48116a93c065474d5" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 36921 } } peers { permanent_uuid: "675f6a806406467fadd299d497d5ef1b" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 36915 } } peers { permanent_uuid: "64c9a129e120464aaca8d82402c8d7e2" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 42967 } }
I20260808 22:22:12.833837 17531 raft_consensus.cc:374] T b4e50a77129f45b19ac8084ba84f3c8f P e91f1d98b12942f48116a93c065474d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:22:12.833932 17531 raft_consensus.cc:729] T b4e50a77129f45b19ac8084ba84f3c8f P e91f1d98b12942f48116a93c065474d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e91f1d98b12942f48116a93c065474d5, State: Initialized, Role: FOLLOWER
I20260808 22:22:12.834383 17531 consensus_queue.cc:260] T b4e50a77129f45b19ac8084ba84f3c8f P e91f1d98b12942f48116a93c065474d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91f1d98b12942f48116a93c065474d5" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 36921 } } peers { permanent_uuid: "675f6a806406467fadd299d497d5ef1b" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 36915 } } peers { permanent_uuid: "64c9a129e120464aaca8d82402c8d7e2" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 42967 } }
I20260808 22:22:12.834501 17538 tablet_bootstrap.cc:492] T b4e50a77129f45b19ac8084ba84f3c8f P 675f6a806406467fadd299d497d5ef1b: No bootstrap required, opened a new log
I20260808 22:22:12.834661 17538 ts_tablet_manager.cc:1397] T b4e50a77129f45b19ac8084ba84f3c8f P 675f6a806406467fadd299d497d5ef1b: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20260808 22:22:12.834507 17532 raft_consensus.cc:348] T b4e50a77129f45b19ac8084ba84f3c8f P 64c9a129e120464aaca8d82402c8d7e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91f1d98b12942f48116a93c065474d5" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 36921 } } peers { permanent_uuid: "675f6a806406467fadd299d497d5ef1b" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 36915 } } peers { permanent_uuid: "64c9a129e120464aaca8d82402c8d7e2" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 42967 } }
I20260808 22:22:12.834751 17532 raft_consensus.cc:374] T b4e50a77129f45b19ac8084ba84f3c8f P 64c9a129e120464aaca8d82402c8d7e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:22:12.834808 17532 raft_consensus.cc:729] T b4e50a77129f45b19ac8084ba84f3c8f P 64c9a129e120464aaca8d82402c8d7e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64c9a129e120464aaca8d82402c8d7e2, State: Initialized, Role: FOLLOWER
I20260808 22:22:12.835201 17532 consensus_queue.cc:260] T b4e50a77129f45b19ac8084ba84f3c8f P 64c9a129e120464aaca8d82402c8d7e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91f1d98b12942f48116a93c065474d5" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 36921 } } peers { permanent_uuid: "675f6a806406467fadd299d497d5ef1b" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 36915 } } peers { permanent_uuid: "64c9a129e120464aaca8d82402c8d7e2" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 42967 } }
I20260808 22:22:12.835400 17531 ts_tablet_manager.cc:1428] T b4e50a77129f45b19ac8084ba84f3c8f P e91f1d98b12942f48116a93c065474d5: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260808 22:22:12.835819 17316 heartbeater.cc:503] Master 127.9.176.254:36427 was elected leader, sending a full tablet report...
I20260808 22:22:12.836061 17532 ts_tablet_manager.cc:1428] T b4e50a77129f45b19ac8084ba84f3c8f P 64c9a129e120464aaca8d82402c8d7e2: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20260808 22:22:12.837007 17121 heartbeater.cc:503] Master 127.9.176.254:36427 was elected leader, sending a full tablet report...
I20260808 22:22:12.838768 17538 raft_consensus.cc:348] T b4e50a77129f45b19ac8084ba84f3c8f P 675f6a806406467fadd299d497d5ef1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91f1d98b12942f48116a93c065474d5" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 36921 } } peers { permanent_uuid: "675f6a806406467fadd299d497d5ef1b" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 36915 } } peers { permanent_uuid: "64c9a129e120464aaca8d82402c8d7e2" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 42967 } }
I20260808 22:22:12.839022 17538 raft_consensus.cc:374] T b4e50a77129f45b19ac8084ba84f3c8f P 675f6a806406467fadd299d497d5ef1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:22:12.839087 17538 raft_consensus.cc:729] T b4e50a77129f45b19ac8084ba84f3c8f P 675f6a806406467fadd299d497d5ef1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 675f6a806406467fadd299d497d5ef1b, State: Initialized, Role: FOLLOWER
I20260808 22:22:12.839476 17538 consensus_queue.cc:260] T b4e50a77129f45b19ac8084ba84f3c8f P 675f6a806406467fadd299d497d5ef1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91f1d98b12942f48116a93c065474d5" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 36921 } } peers { permanent_uuid: "675f6a806406467fadd299d497d5ef1b" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 36915 } } peers { permanent_uuid: "64c9a129e120464aaca8d82402c8d7e2" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 42967 } }
I20260808 22:22:12.840102 17509 heartbeater.cc:503] Master 127.9.176.254:36427 was elected leader, sending a full tablet report...
I20260808 22:22:12.841295 17538 ts_tablet_manager.cc:1428] T b4e50a77129f45b19ac8084ba84f3c8f P 675f6a806406467fadd299d497d5ef1b: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
W20260808 22:22:12.854285 17317 tablet.cc:2369] T b4e50a77129f45b19ac8084ba84f3c8f P 64c9a129e120464aaca8d82402c8d7e2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 22:22:12.936743 17122 tablet.cc:2369] T b4e50a77129f45b19ac8084ba84f3c8f P e91f1d98b12942f48116a93c065474d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 22:22:12.951617 17548 raft_consensus.cc:482] T b4e50a77129f45b19ac8084ba84f3c8f P 64c9a129e120464aaca8d82402c8d7e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 22:22:12.951813 17548 raft_consensus.cc:504] T b4e50a77129f45b19ac8084ba84f3c8f P 64c9a129e120464aaca8d82402c8d7e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91f1d98b12942f48116a93c065474d5" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 36921 } } peers { permanent_uuid: "675f6a806406467fadd299d497d5ef1b" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 36915 } } peers { permanent_uuid: "64c9a129e120464aaca8d82402c8d7e2" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 42967 } }
I20260808 22:22:12.952942 17548 leader_election.cc:288] T b4e50a77129f45b19ac8084ba84f3c8f P 64c9a129e120464aaca8d82402c8d7e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e91f1d98b12942f48116a93c065474d5 (127.9.176.193:36921), 675f6a806406467fadd299d497d5ef1b (127.9.176.195:36915)
I20260808 22:22:12.954656 17528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:12.953392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.194:56151 (local address 127.9.176.195:36915)
0808 22:22:12.953516 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:12.953520 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:12.953566 (+ 46us) server_negotiation.cc:407] Connection header received
0808 22:22:12.953662 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:12.953665 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:12.953710 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:12.953781 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:12.954341 (+ 560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:12.954345 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:12.954347 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:22:12.954359 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:12.954377 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:22:12.954387 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:12.954465 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:12.954523 (+ 58us) server_negotiation.cc:299] Negotiation successful
0808 22:22:12.954555 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 22:22:12.954655 17526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:12.953236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.194:57629 (local address 127.9.176.193:36921)
0808 22:22:12.953354 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:12.953358 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:12.953467 (+ 109us) server_negotiation.cc:407] Connection header received
0808 22:22:12.953524 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:12.953528 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:12.953570 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:12.953658 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:12.954276 (+ 618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:12.954280 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:12.954283 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:22:12.954297 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:12.954321 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:22:12.954331 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:12.954421 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:12.954480 (+ 59us) server_negotiation.cc:299] Negotiation successful
0808 22:22:12.954523 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260808 22:22:12.954685 17550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:12.953171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.195:36915 (local address 127.9.176.194:56151)
0808 22:22:12.953497 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:12.953509 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:12.953579 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:12.953812 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:12.953814 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:12.953827 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:12.954194 (+ 367us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 22:22:12.954196 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:12.954204 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 22:22:12.954226 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:22:12.954455 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:22:12.954458 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:22:12.954461 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 22:22:12.954521 (+ 60us) client_negotiation.cc:240] Negotiation successful
0808 22:22:12.954545 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"mutex_wait_us":3,"thread_start_us":120,"threads_started":1}
I20260808 22:22:12.954654 17318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:12.953146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.193:36921 (local address 127.9.176.194:57629)
0808 22:22:12.953337 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:12.953351 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:12.953440 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:12.953682 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:12.953684 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:12.953697 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:12.954109 (+ 412us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 22:22:12.954111 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:12.954123 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 22:22:12.954167 (+ 44us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:22:12.954405 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:22:12.954407 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:22:12.954410 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 22:22:12.954482 (+ 72us) client_negotiation.cc:240] Negotiation successful
0808 22:22:12.954518 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106}
I20260808 22:22:12.955312 17053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4e50a77129f45b19ac8084ba84f3c8f" candidate_uuid: "64c9a129e120464aaca8d82402c8d7e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e91f1d98b12942f48116a93c065474d5" is_pre_election: true
I20260808 22:22:12.955503 17447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4e50a77129f45b19ac8084ba84f3c8f" candidate_uuid: "64c9a129e120464aaca8d82402c8d7e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "675f6a806406467fadd299d497d5ef1b" is_pre_election: true
I20260808 22:22:12.955590 17053 raft_consensus.cc:2454] T b4e50a77129f45b19ac8084ba84f3c8f P e91f1d98b12942f48116a93c065474d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64c9a129e120464aaca8d82402c8d7e2 in term 0.
I20260808 22:22:12.955767 17447 raft_consensus.cc:2454] T b4e50a77129f45b19ac8084ba84f3c8f P 675f6a806406467fadd299d497d5ef1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64c9a129e120464aaca8d82402c8d7e2 in term 0.
I20260808 22:22:12.956050 17157 leader_election.cc:302] T b4e50a77129f45b19ac8084ba84f3c8f P 64c9a129e120464aaca8d82402c8d7e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64c9a129e120464aaca8d82402c8d7e2, e91f1d98b12942f48116a93c065474d5; no voters:
I20260808 22:22:12.956321 17548 raft_consensus.cc:2781] T b4e50a77129f45b19ac8084ba84f3c8f P 64c9a129e120464aaca8d82402c8d7e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 22:22:12.956405 17548 raft_consensus.cc:482] T b4e50a77129f45b19ac8084ba84f3c8f P 64c9a129e120464aaca8d82402c8d7e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 22:22:12.956453 17548 raft_consensus.cc:3037] T b4e50a77129f45b19ac8084ba84f3c8f P 64c9a129e120464aaca8d82402c8d7e2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:22:12.957494 17548 raft_consensus.cc:504] T b4e50a77129f45b19ac8084ba84f3c8f P 64c9a129e120464aaca8d82402c8d7e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91f1d98b12942f48116a93c065474d5" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 36921 } } peers { permanent_uuid: "675f6a806406467fadd299d497d5ef1b" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 36915 } } peers { permanent_uuid: "64c9a129e120464aaca8d82402c8d7e2" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 42967 } }
I20260808 22:22:12.957800 17548 leader_election.cc:288] T b4e50a77129f45b19ac8084ba84f3c8f P 64c9a129e120464aaca8d82402c8d7e2 [CANDIDATE]: Term 1 election: Requested vote from peers e91f1d98b12942f48116a93c065474d5 (127.9.176.193:36921), 675f6a806406467fadd299d497d5ef1b (127.9.176.195:36915)
I20260808 22:22:12.958107 17447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4e50a77129f45b19ac8084ba84f3c8f" candidate_uuid: "64c9a129e120464aaca8d82402c8d7e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "675f6a806406467fadd299d497d5ef1b"
I20260808 22:22:12.958122 17053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4e50a77129f45b19ac8084ba84f3c8f" candidate_uuid: "64c9a129e120464aaca8d82402c8d7e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e91f1d98b12942f48116a93c065474d5"
I20260808 22:22:12.958235 17053 raft_consensus.cc:3037] T b4e50a77129f45b19ac8084ba84f3c8f P e91f1d98b12942f48116a93c065474d5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:22:12.958235 17447 raft_consensus.cc:3037] T b4e50a77129f45b19ac8084ba84f3c8f P 675f6a806406467fadd299d497d5ef1b [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:22:12.959471 17053 raft_consensus.cc:2454] T b4e50a77129f45b19ac8084ba84f3c8f P e91f1d98b12942f48116a93c065474d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64c9a129e120464aaca8d82402c8d7e2 in term 1.
I20260808 22:22:12.959471 17447 raft_consensus.cc:2454] T b4e50a77129f45b19ac8084ba84f3c8f P 675f6a806406467fadd299d497d5ef1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64c9a129e120464aaca8d82402c8d7e2 in term 1.
I20260808 22:22:12.959762 17157 leader_election.cc:302] T b4e50a77129f45b19ac8084ba84f3c8f P 64c9a129e120464aaca8d82402c8d7e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 64c9a129e120464aaca8d82402c8d7e2, 675f6a806406467fadd299d497d5ef1b, e91f1d98b12942f48116a93c065474d5; no voters:
I20260808 22:22:12.959924 17548 raft_consensus.cc:2781] T b4e50a77129f45b19ac8084ba84f3c8f P 64c9a129e120464aaca8d82402c8d7e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:22:12.960117 17548 raft_consensus.cc:686] T b4e50a77129f45b19ac8084ba84f3c8f P 64c9a129e120464aaca8d82402c8d7e2 [term 1 LEADER]: Becoming Leader. State: Replica: 64c9a129e120464aaca8d82402c8d7e2, State: Running, Role: LEADER
I20260808 22:22:12.960445 17548 consensus_queue.cc:237] T b4e50a77129f45b19ac8084ba84f3c8f P 64c9a129e120464aaca8d82402c8d7e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91f1d98b12942f48116a93c065474d5" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 36921 } } peers { permanent_uuid: "675f6a806406467fadd299d497d5ef1b" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 36915 } } peers { permanent_uuid: "64c9a129e120464aaca8d82402c8d7e2" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 42967 } }
I20260808 22:22:12.963840 16838 catalog_manager.cc:5495] T b4e50a77129f45b19ac8084ba84f3c8f P 64c9a129e120464aaca8d82402c8d7e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 64c9a129e120464aaca8d82402c8d7e2 (127.9.176.194). New cstate: current_term: 1 leader_uuid: "64c9a129e120464aaca8d82402c8d7e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91f1d98b12942f48116a93c065474d5" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 36921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "675f6a806406467fadd299d497d5ef1b" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 36915 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64c9a129e120464aaca8d82402c8d7e2" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 42967 } health_report { overall_health: HEALTHY } } }
I20260808 22:22:13.004709 17530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:13.003259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50928 (local address 127.9.176.194:42967)
0808 22:22:13.003373 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:13.003378 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:13.003391 (+ 13us) server_negotiation.cc:407] Connection header received
0808 22:22:13.003536 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:13.003539 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:13.003598 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:13.003666 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:13.004329 (+ 663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:13.004333 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:13.004335 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:22:13.004347 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:13.004368 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:22:13.004377 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:13.004448 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:13.004520 (+ 72us) server_negotiation.cc:299] Negotiation successful
0808 22:22:13.004560 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260808 22:22:13.010248 17053 raft_consensus.cc:1261] T b4e50a77129f45b19ac8084ba84f3c8f P e91f1d98b12942f48116a93c065474d5 [term 1 FOLLOWER]: Refusing update from remote peer 64c9a129e120464aaca8d82402c8d7e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:22:13.010249 17447 raft_consensus.cc:1261] T b4e50a77129f45b19ac8084ba84f3c8f P 675f6a806406467fadd299d497d5ef1b [term 1 FOLLOWER]: Refusing update from remote peer 64c9a129e120464aaca8d82402c8d7e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:22:13.010931 17548 consensus_queue.cc:1035] T b4e50a77129f45b19ac8084ba84f3c8f P 64c9a129e120464aaca8d82402c8d7e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "675f6a806406467fadd299d497d5ef1b" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 36915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:22:13.011109 17555 consensus_queue.cc:1035] T b4e50a77129f45b19ac8084ba84f3c8f P 64c9a129e120464aaca8d82402c8d7e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e91f1d98b12942f48116a93c065474d5" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 36921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:22:13.018297 17558 mvcc.cc:205] Tried to move back new op lower bound from 7316388794403225600 to 7316388794208374784. Current Snapshot: MvccSnapshot[applied={T|T < 7316388794403225600}]
I20260808 22:22:13.029404 16838 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54758:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:22:13.029652 16838 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54758:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:22:13.032146 16838 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 35e9c6db6ceb49dda7d4ea0c1fb044cd: Sending DeleteTablet for 3 replicas of tablet b4e50a77129f45b19ac8084ba84f3c8f
I20260808 22:22:13.032979 17419 tablet_service.cc:1514] Processing DeleteTablet for tablet b4e50a77129f45b19ac8084ba84f3c8f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:22:13 UTC) from {username='slave'} at 127.0.0.1:40402
I20260808 22:22:13.032989 17023 tablet_service.cc:1514] Processing DeleteTablet for tablet b4e50a77129f45b19ac8084ba84f3c8f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:22:13 UTC) from {username='slave'} at 127.0.0.1:54516
I20260808 22:22:13.033167 17219 tablet_service.cc:1514] Processing DeleteTablet for tablet b4e50a77129f45b19ac8084ba84f3c8f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:22:13 UTC) from {username='slave'} at 127.0.0.1:50914
I20260808 22:22:13.033586 17575 tablet_replica.cc:330] T b4e50a77129f45b19ac8084ba84f3c8f P 64c9a129e120464aaca8d82402c8d7e2: stopping tablet replica
I20260808 22:22:13.033679 17572 tablet_replica.cc:330] T b4e50a77129f45b19ac8084ba84f3c8f P 675f6a806406467fadd299d497d5ef1b: stopping tablet replica
I20260808 22:22:13.033900 17575 raft_consensus.cc:2229] T b4e50a77129f45b19ac8084ba84f3c8f P 64c9a129e120464aaca8d82402c8d7e2 [term 1 LEADER]: Raft consensus shutting down.
I20260808 22:22:13.034125 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 16932
I20260808 22:22:13.034082 17572 raft_consensus.cc:2229] T b4e50a77129f45b19ac8084ba84f3c8f P 675f6a806406467fadd299d497d5ef1b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:22:13.034286 17575 raft_consensus.cc:2258] T b4e50a77129f45b19ac8084ba84f3c8f P 64c9a129e120464aaca8d82402c8d7e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:22:13.040530 17575 ts_tablet_manager.cc:1905] T b4e50a77129f45b19ac8084ba84f3c8f P 64c9a129e120464aaca8d82402c8d7e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:22:13.043262 17575 ts_tablet_manager.cc:1918] T b4e50a77129f45b19ac8084ba84f3c8f P 64c9a129e120464aaca8d82402c8d7e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 22:22:13.043360 17575 log.cc:1196] T b4e50a77129f45b19ac8084ba84f3c8f P 64c9a129e120464aaca8d82402c8d7e2: Deleting WAL directory at /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786227706282457-9923-0/minicluster-data/ts-1/wal/wals/b4e50a77129f45b19ac8084ba84f3c8f
I20260808 22:22:13.043746 17575 ts_tablet_manager.cc:1939] T b4e50a77129f45b19ac8084ba84f3c8f P 64c9a129e120464aaca8d82402c8d7e2: Deleting consensus metadata
I20260808 22:22:13.044256 17572 raft_consensus.cc:2258] T b4e50a77129f45b19ac8084ba84f3c8f P 675f6a806406467fadd299d497d5ef1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:22:13.044939 16814 catalog_manager.cc:4841] TS 64c9a129e120464aaca8d82402c8d7e2 (127.9.176.194:42967): tablet b4e50a77129f45b19ac8084ba84f3c8f (table test-table [id=524d995258b74c9083bbb925e871bdab]) successfully deleted
W20260808 22:22:13.049695 16815 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.176.193:36921 (error 108)
I20260808 22:22:13.050251 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 17128
I20260808 22:22:13.052448 17527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:13.050625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.193:36921 (local address 127.0.0.1:54522)
0808 22:22:13.052284 (+ 1659us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:13.052361 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.193:36921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1570}
W20260808 22:22:13.053746 16815 catalog_manager.cc:4584] TS e91f1d98b12942f48116a93c065474d5 (127.9.176.193:36921): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b4e50a77129f45b19ac8084ba84f3c8f: Network error: Client connection negotiation failed: client connection to 127.9.176.193:36921: connect: Connection refused (error 111)
I20260808 22:22:13.058667 17572 ts_tablet_manager.cc:1905] T b4e50a77129f45b19ac8084ba84f3c8f P 675f6a806406467fadd299d497d5ef1b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:22:13.060881 17572 ts_tablet_manager.cc:1918] T b4e50a77129f45b19ac8084ba84f3c8f P 675f6a806406467fadd299d497d5ef1b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 22:22:13.061017 17572 log.cc:1196] T b4e50a77129f45b19ac8084ba84f3c8f P 675f6a806406467fadd299d497d5ef1b: Deleting WAL directory at /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786227706282457-9923-0/minicluster-data/ts-2/wal/wals/b4e50a77129f45b19ac8084ba84f3c8f
I20260808 22:22:13.061262 17572 ts_tablet_manager.cc:1939] T b4e50a77129f45b19ac8084ba84f3c8f P 675f6a806406467fadd299d497d5ef1b: Deleting consensus metadata
I20260808 22:22:13.062436 16817 catalog_manager.cc:4841] TS 675f6a806406467fadd299d497d5ef1b (127.9.176.195:36915): tablet b4e50a77129f45b19ac8084ba84f3c8f (table test-table [id=524d995258b74c9083bbb925e871bdab]) successfully deleted
I20260808 22:22:13.063140 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 17319
I20260808 22:22:13.073251 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 16793
2026-08-08T22:22:13Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1052 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260808 22:22:13.104123 9923 mini_oidc.cc:152] Starting JWKS server
I20260808 22:22:13.106065 9923 webserver.cc:460] Webserver started at https://127.0.0.1:46291/ using document root <none> and password file <none>
I20260808 22:22:13.106231 9923 mini_oidc.cc:189] Starting OIDC Discovery server
I20260808 22:22:13.106484 9923 webserver.cc:460] Webserver started at http://127.0.0.1:44563/ using document root <none> and password file <none>
2026-08-08T22:22:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:22:13Z Disabled control of system clock
I20260808 22:22:13.115787 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786227706282457-9923-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786227706282457-9923-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786227706282457-9923-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:46699
--webserver_interface=127.9.176.254
--webserver_port=0
--builtin_ntp_servers=127.9.176.212:39415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.176.254:46699
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46291/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786227706282457-9923-0/testchainca.cert with env {}
W20260808 22:22:13.225579 17582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:22:13.225843 17582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:22:13.225883 17582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:22:13.229835 17582 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:22:13.229890 17582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:22:13.229904 17582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:22:13.229919 17582 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:22:13.229939 17582 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260808 22:22:13.229954 17582 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260808 22:22:13.229970 17582 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46291/jwks.json
I20260808 22:22:13.235169 17582 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:39415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786227706282457-9923-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786227706282457-9923-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.176.254:46699
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:46699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46291/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.176.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786227706282457-9923-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786227706282457-9923-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:22:13.236132 17582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:22:13.237155 17582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:22:13.245254 17589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:13.245254 17590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:13.245247 17593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:22:13.245464 17582 server_base.cc:1034] running on GCE node
I20260808 22:22:13.246110 17582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:22:13.247159 17582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:22:13.248327 17582 hybrid_clock.cc:648] HybridClock initialized: now 1786227733248307 us; error 30 us; skew 500 ppm
I20260808 22:22:13.250154 17582 webserver.cc:460] Webserver started at http://127.9.176.254:44059/ using document root <none> and password file <none>
I20260808 22:22:13.250720 17582 fs_manager.cc:359] Metadata directory not provided
I20260808 22:22:13.250788 17582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:22:13.250976 17582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:22:13.253075 17582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786227706282457-9923-0/minicluster-data/master-0/data/instance:
uuid: "319ddf5288f24de38b328524bd736578"
format_stamp: "Formatted at 2026-08-08 22:22:13 on dist-test-slave-ncp9"
server_key: "4a2e27a9a00552cb78c1c40349042e9d"
server_key_iv: "c3d2d8e7894bbbdb0ef58ee4f5edd620"
server_key_version: "encryptionkey@0"
I20260808 22:22:13.253607 17582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786227706282457-9923-0/minicluster-data/master-0/wal/instance:
uuid: "319ddf5288f24de38b328524bd736578"
format_stamp: "Formatted at 2026-08-08 22:22:13 on dist-test-slave-ncp9"
server_key: "4a2e27a9a00552cb78c1c40349042e9d"
server_key_iv: "c3d2d8e7894bbbdb0ef58ee4f5edd620"
server_key_version: "encryptionkey@0"
I20260808 22:22:13.257129 17582 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260808 22:22:13.259346 17602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:13.260478 17582 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 22:22:13.260604 17582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786227706282457-9923-0/minicluster-data/master-0/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786227706282457-9923-0/minicluster-data/master-0/wal
uuid: "319ddf5288f24de38b328524bd736578"
format_stamp: "Formatted at 2026-08-08 22:22:13 on dist-test-slave-ncp9"
server_key: "4a2e27a9a00552cb78c1c40349042e9d"
server_key_iv: "c3d2d8e7894bbbdb0ef58ee4f5edd620"
server_key_version: "encryptionkey@0"
I20260808 22:22:13.260696 17582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786227706282457-9923-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786227706282457-9923-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786227706282457-9923-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:22:13.275144 17582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:22:13.275759 17582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:22:13.275928 17582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:22:13.282560 17582 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.254:46699
I20260808 22:22:13.282579 17684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.254:46699 every 8 connection(s)
I20260808 22:22:13.283455 17582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
I20260808 22:22:13.286181 17688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:22:13.291182 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 17582
I20260808 22:22:13.291307 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786227706282457-9923-0/minicluster-data/master-0/wal/instance
I20260808 22:22:13.291599 9923 external_mini_cluster.cc:1428] Setting key 60040d838a2f78e152ebee29632e04b7
I20260808 22:22:13.291612 17688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 319ddf5288f24de38b328524bd736578: Bootstrap starting.
I20260808 22:22:13.294148 17688 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 319ddf5288f24de38b328524bd736578: Neither blocks nor log segments found. Creating new log.
I20260808 22:22:13.294859 17688 log.cc:824] T 00000000000000000000000000000000 P 319ddf5288f24de38b328524bd736578: Log is configured to *not* fsync() on all Append() calls
I20260808 22:22:13.296598 17688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 319ddf5288f24de38b328524bd736578: No bootstrap required, opened a new log
I20260808 22:22:13.299990 17688 raft_consensus.cc:348] T 00000000000000000000000000000000 P 319ddf5288f24de38b328524bd736578 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "319ddf5288f24de38b328524bd736578" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 46699 } }
I20260808 22:22:13.300205 17688 raft_consensus.cc:374] T 00000000000000000000000000000000 P 319ddf5288f24de38b328524bd736578 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:22:13.300262 17688 raft_consensus.cc:729] T 00000000000000000000000000000000 P 319ddf5288f24de38b328524bd736578 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 319ddf5288f24de38b328524bd736578, State: Initialized, Role: FOLLOWER
I20260808 22:22:13.300385 17692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:13.292824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36652 (local address 127.9.176.254:46699)
0808 22:22:13.293482 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:13.293499 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:13.293533 (+ 34us) server_negotiation.cc:407] Connection header received
0808 22:22:13.294155 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:13.294171 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:13.294577 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:13.294885 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:13.295291 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:13.296278 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:13.297300 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:13.297795 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:13.297949 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:13.298717 (+ 768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:13.298721 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:13.298733 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:22:13.299097 (+ 364us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:13.299154 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:22:13.299175 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:13.299263 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:13.299402 (+ 139us) server_negotiation.cc:299] Negotiation successful
0808 22:22:13.299643 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":139,"threads_started":1}
I20260808 22:22:13.300714 17688 consensus_queue.cc:260] T 00000000000000000000000000000000 P 319ddf5288f24de38b328524bd736578 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "319ddf5288f24de38b328524bd736578" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 46699 } }
I20260808 22:22:13.300856 17688 raft_consensus.cc:388] T 00000000000000000000000000000000 P 319ddf5288f24de38b328524bd736578 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:22:13.300943 17688 raft_consensus.cc:482] T 00000000000000000000000000000000 P 319ddf5288f24de38b328524bd736578 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:22:13.301041 17688 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 319ddf5288f24de38b328524bd736578 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:22:13.302008 17688 raft_consensus.cc:504] T 00000000000000000000000000000000 P 319ddf5288f24de38b328524bd736578 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "319ddf5288f24de38b328524bd736578" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 46699 } }
I20260808 22:22:13.302373 17688 leader_election.cc:302] T 00000000000000000000000000000000 P 319ddf5288f24de38b328524bd736578 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 319ddf5288f24de38b328524bd736578; no voters:
I20260808 22:22:13.302668 17688 leader_election.cc:288] T 00000000000000000000000000000000 P 319ddf5288f24de38b328524bd736578 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:22:13.302742 17695 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 319ddf5288f24de38b328524bd736578 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:22:13.302928 17695 raft_consensus.cc:686] T 00000000000000000000000000000000 P 319ddf5288f24de38b328524bd736578 [term 1 LEADER]: Becoming Leader. State: Replica: 319ddf5288f24de38b328524bd736578, State: Running, Role: LEADER
I20260808 22:22:13.303223 17695 consensus_queue.cc:237] T 00000000000000000000000000000000 P 319ddf5288f24de38b328524bd736578 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "319ddf5288f24de38b328524bd736578" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 46699 } }
I20260808 22:22:13.303376 9923 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 22:22:13.303916 17688 sys_catalog.cc:564] T 00000000000000000000000000000000 P 319ddf5288f24de38b328524bd736578 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:22:13.304253 17695 sys_catalog.cc:455] T 00000000000000000000000000000000 P 319ddf5288f24de38b328524bd736578 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 319ddf5288f24de38b328524bd736578. Latest consensus state: current_term: 1 leader_uuid: "319ddf5288f24de38b328524bd736578" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "319ddf5288f24de38b328524bd736578" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 46699 } } }
I20260808 22:22:13.304373 17695 sys_catalog.cc:458] T 00000000000000000000000000000000 P 319ddf5288f24de38b328524bd736578 [sys.catalog]: This master's current role is: LEADER
I20260808 22:22:13.304654 17696 sys_catalog.cc:455] T 00000000000000000000000000000000 P 319ddf5288f24de38b328524bd736578 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "319ddf5288f24de38b328524bd736578" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "319ddf5288f24de38b328524bd736578" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 46699 } } }
I20260808 22:22:13.304733 17696 sys_catalog.cc:458] T 00000000000000000000000000000000 P 319ddf5288f24de38b328524bd736578 [sys.catalog]: This master's current role is: LEADER
W20260808 22:22:13.308668 17713 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 319ddf5288f24de38b328524bd736578: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 22:22:13.308743 17713 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 22:22:13.308848 17700 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:22:13.309794 17700 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:22:13.314867 17700 catalog_manager.cc:1347] Generated new cluster ID: 4b3a86f8e07d4a049bd70bfd25242b94
I20260808 22:22:13.314934 17700 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:22:13.322523 17700 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:22:13.323412 17700 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:22:13.327798 17700 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 319ddf5288f24de38b328524bd736578: Generated new TSK 0
I20260808 22:22:13.328357 17700 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260808 22:22:13.358963 17730 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260808 22:22:13.359115 17730 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260808 22:22:13.360198 17692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:13.342338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36654 (local address 127.9.176.254:46699)
0808 22:22:13.342458 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:13.342463 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:13.342671 (+ 208us) server_negotiation.cc:407] Connection header received
0808 22:22:13.342850 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:13.342854 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:13.342908 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:13.343012 (+ 104us) server_negotiation.cc:226] Negotiated authn=JWT
0808 22:22:13.343283 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:13.344081 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:13.344944 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:13.345303 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:13.345402 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:13.345684 (+ 282us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0808 22:22:13.359408 (+ 13724us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46291/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0808 22:22:13.359678 (+ 270us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36654: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46291/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":45}
W20260808 22:22:13.360267 17692 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36654: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46291/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260808 22:22:13.361191 17729 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:22:13.342236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:46699 (local address 127.0.0.1:36654)
0808 22:22:13.342598 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:13.342610 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:13.342712 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:13.343062 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:13.343065 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:13.343073 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0808 22:22:13.343184 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:13.343189 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:13.344206 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:13.344208 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:13.344822 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:13.344830 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:13.345403 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:13.345406 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:13.345475 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:13.345517 (+ 42us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0808 22:22:13.359609 (+ 14092us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46291/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0808 22:22:13.359803 (+ 194us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.9.176.254:46699: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46291/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":105,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.9.176.254:46699: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46291/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260808 22:22:13.362120 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 17582
2026-08-08T22:22:13Z chronyd exiting
I20260808 22:22:13.406279 9923 test_util.cc:175] -----------------------------------------------
I20260808 22:22:13.406368 9923 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786227706282457-9923-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (303 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260808 22:22:13.407445 9923 mini_oidc.cc:152] Starting JWKS server
I20260808 22:22:13.408275 9923 webserver.cc:460] Webserver started at https://127.0.0.1:34275/ using document root <none> and password file <none>
I20260808 22:22:13.408360 9923 mini_oidc.cc:189] Starting OIDC Discovery server
I20260808 22:22:13.408591 9923 webserver.cc:460] Webserver started at http://127.0.0.1:34905/ using document root <none> and password file <none>
2026-08-08T22:22:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:22:13Z Disabled control of system clock
I20260808 22:22:13.418038 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786227706282457-9923-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786227706282457-9923-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786227706282457-9923-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:43767
--webserver_interface=127.9.176.254
--webserver_port=0
--builtin_ntp_servers=127.9.176.212:38819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.176.254:43767
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34275/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786227706282457-9923-0/testchainca.cert with env {}
W20260808 22:22:13.538244 17738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:22:13.538503 17738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:22:13.538540 17738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:22:13.542502 17738 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:22:13.542553 17738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:22:13.542567 17738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:22:13.542588 17738 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:22:13.542603 17738 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260808 22:22:13.542624 17738 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260808 22:22:13.542644 17738 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34275/jwks.json
I20260808 22:22:13.547863 17738 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:38819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786227706282457-9923-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786227706282457-9923-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.176.254:43767
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:43767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34275/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.176.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786227706282457-9923-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17738
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786227706282457-9923-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:22:13.548951 17738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:22:13.549922 17738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:22:13.558167 17748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:13.558190 17752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:13.558185 17749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:22:13.558358 17738 server_base.cc:1034] running on GCE node
I20260808 22:22:13.558991 17738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:22:13.560034 17738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:22:13.561199 17738 hybrid_clock.cc:648] HybridClock initialized: now 1786227733561184 us; error 27 us; skew 500 ppm
I20260808 22:22:13.562951 17738 webserver.cc:460] Webserver started at http://127.9.176.254:34751/ using document root <none> and password file <none>
I20260808 22:22:13.563508 17738 fs_manager.cc:359] Metadata directory not provided
I20260808 22:22:13.563575 17738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:22:13.563763 17738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:22:13.565940 17738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786227706282457-9923-0/minicluster-data/master-0/data/instance:
uuid: "197c02abd275423b8d26bf997e9fb663"
format_stamp: "Formatted at 2026-08-08 22:22:13 on dist-test-slave-ncp9"
server_key: "74f1556b0fb3f5efa796dbf6ec7b09c0"
server_key_iv: "33863e785ec237118b27e808f82bb146"
server_key_version: "encryptionkey@0"
I20260808 22:22:13.566524 17738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786227706282457-9923-0/minicluster-data/master-0/wal/instance:
uuid: "197c02abd275423b8d26bf997e9fb663"
format_stamp: "Formatted at 2026-08-08 22:22:13 on dist-test-slave-ncp9"
server_key: "74f1556b0fb3f5efa796dbf6ec7b09c0"
server_key_iv: "33863e785ec237118b27e808f82bb146"
server_key_version: "encryptionkey@0"
I20260808 22:22:13.570253 17738 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 22:22:13.572708 17763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:13.573944 17738 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 22:22:13.574062 17738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786227706282457-9923-0/minicluster-data/master-0/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786227706282457-9923-0/minicluster-data/master-0/wal
uuid: "197c02abd275423b8d26bf997e9fb663"
format_stamp: "Formatted at 2026-08-08 22:22:13 on dist-test-slave-ncp9"
server_key: "74f1556b0fb3f5efa796dbf6ec7b09c0"
server_key_iv: "33863e785ec237118b27e808f82bb146"
server_key_version: "encryptionkey@0"
I20260808 22:22:13.574172 17738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786227706282457-9923-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786227706282457-9923-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786227706282457-9923-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:22:13.590791 17738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:22:13.591463 17738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:22:13.591638 17738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:22:13.598115 17738 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.254:43767
I20260808 22:22:13.598129 17835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.254:43767 every 8 connection(s)
I20260808 22:22:13.599001 17738 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
I20260808 22:22:13.601944 17837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:22:13.603221 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 17738
I20260808 22:22:13.603328 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786227706282457-9923-0/minicluster-data/master-0/wal/instance
I20260808 22:22:13.603583 9923 external_mini_cluster.cc:1428] Setting key 5edb7f412599dfc58dbcf1dcc65123ea
I20260808 22:22:13.608322 17837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 197c02abd275423b8d26bf997e9fb663: Bootstrap starting.
I20260808 22:22:13.610879 17837 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 197c02abd275423b8d26bf997e9fb663: Neither blocks nor log segments found. Creating new log.
I20260808 22:22:13.611645 17837 log.cc:824] T 00000000000000000000000000000000 P 197c02abd275423b8d26bf997e9fb663: Log is configured to *not* fsync() on all Append() calls
I20260808 22:22:13.613698 17843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:13.604872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43148 (local address 127.9.176.254:43767)
0808 22:22:13.605551 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:13.605572 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:13.605616 (+ 44us) server_negotiation.cc:407] Connection header received
0808 22:22:13.606458 (+ 842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:13.606481 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:13.606985 (+ 504us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:13.607354 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:13.607793 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:13.609265 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:13.610139 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:13.610820 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:13.610982 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:13.611726 (+ 744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:13.611732 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:13.611748 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:22:13.612189 (+ 441us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:13.612245 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:22:13.612267 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:13.612349 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:13.612504 (+ 155us) server_negotiation.cc:299] Negotiation successful
0808 22:22:13.612769 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":103,"threads_started":1}
I20260808 22:22:13.613801 17837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 197c02abd275423b8d26bf997e9fb663: No bootstrap required, opened a new log
I20260808 22:22:13.616460 9923 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 22:22:13.617674 17837 raft_consensus.cc:348] T 00000000000000000000000000000000 P 197c02abd275423b8d26bf997e9fb663 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "197c02abd275423b8d26bf997e9fb663" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 43767 } }
I20260808 22:22:13.617890 17837 raft_consensus.cc:374] T 00000000000000000000000000000000 P 197c02abd275423b8d26bf997e9fb663 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:22:13.617939 17837 raft_consensus.cc:729] T 00000000000000000000000000000000 P 197c02abd275423b8d26bf997e9fb663 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 197c02abd275423b8d26bf997e9fb663, State: Initialized, Role: FOLLOWER
I20260808 22:22:13.618316 17837 consensus_queue.cc:260] T 00000000000000000000000000000000 P 197c02abd275423b8d26bf997e9fb663 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "197c02abd275423b8d26bf997e9fb663" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 43767 } }
I20260808 22:22:13.618436 17837 raft_consensus.cc:388] T 00000000000000000000000000000000 P 197c02abd275423b8d26bf997e9fb663 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:22:13.618496 17837 raft_consensus.cc:482] T 00000000000000000000000000000000 P 197c02abd275423b8d26bf997e9fb663 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:22:13.618584 17837 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 197c02abd275423b8d26bf997e9fb663 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:22:13.619441 17837 raft_consensus.cc:504] T 00000000000000000000000000000000 P 197c02abd275423b8d26bf997e9fb663 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "197c02abd275423b8d26bf997e9fb663" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 43767 } }
I20260808 22:22:13.619757 17837 leader_election.cc:302] T 00000000000000000000000000000000 P 197c02abd275423b8d26bf997e9fb663 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 197c02abd275423b8d26bf997e9fb663; no voters:
I20260808 22:22:13.620026 17837 leader_election.cc:288] T 00000000000000000000000000000000 P 197c02abd275423b8d26bf997e9fb663 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:22:13.620121 17845 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 197c02abd275423b8d26bf997e9fb663 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:22:13.620321 17845 raft_consensus.cc:686] T 00000000000000000000000000000000 P 197c02abd275423b8d26bf997e9fb663 [term 1 LEADER]: Becoming Leader. State: Replica: 197c02abd275423b8d26bf997e9fb663, State: Running, Role: LEADER
I20260808 22:22:13.620633 17845 consensus_queue.cc:237] T 00000000000000000000000000000000 P 197c02abd275423b8d26bf997e9fb663 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "197c02abd275423b8d26bf997e9fb663" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 43767 } }
I20260808 22:22:13.621187 17837 sys_catalog.cc:564] T 00000000000000000000000000000000 P 197c02abd275423b8d26bf997e9fb663 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:22:13.622234 17849 sys_catalog.cc:455] T 00000000000000000000000000000000 P 197c02abd275423b8d26bf997e9fb663 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 197c02abd275423b8d26bf997e9fb663. Latest consensus state: current_term: 1 leader_uuid: "197c02abd275423b8d26bf997e9fb663" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "197c02abd275423b8d26bf997e9fb663" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 43767 } } }
I20260808 22:22:13.622498 17846 sys_catalog.cc:455] T 00000000000000000000000000000000 P 197c02abd275423b8d26bf997e9fb663 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "197c02abd275423b8d26bf997e9fb663" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "197c02abd275423b8d26bf997e9fb663" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 43767 } } }
I20260808 22:22:13.622615 17846 sys_catalog.cc:458] T 00000000000000000000000000000000 P 197c02abd275423b8d26bf997e9fb663 [sys.catalog]: This master's current role is: LEADER
I20260808 22:22:13.622584 17849 sys_catalog.cc:458] T 00000000000000000000000000000000 P 197c02abd275423b8d26bf997e9fb663 [sys.catalog]: This master's current role is: LEADER
W20260808 22:22:13.625623 17868 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 197c02abd275423b8d26bf997e9fb663: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 22:22:13.625686 17868 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 22:22:13.625799 17864 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:22:13.626693 17864 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:22:13.631842 17864 catalog_manager.cc:1347] Generated new cluster ID: 4ccd5dba049948b58868b47762a3901e
I20260808 22:22:13.631906 17864 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:22:13.649269 17864 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:22:13.650115 17864 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:22:13.654279 17864 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 197c02abd275423b8d26bf997e9fb663: Generated new TSK 0
I20260808 22:22:13.654887 17864 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260808 22:22:13.685113 17883 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260808 22:22:13.685195 17883 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260808 22:22:13.685739 17843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:13.662952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43162 (local address 127.9.176.254:43767)
0808 22:22:13.663080 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:13.663086 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:13.663255 (+ 169us) server_negotiation.cc:407] Connection header received
0808 22:22:13.663404 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:13.663406 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:13.663450 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:13.663534 (+ 84us) server_negotiation.cc:226] Negotiated authn=JWT
0808 22:22:13.663759 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:13.664291 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:13.665493 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:13.665704 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:13.665774 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:13.666059 (+ 285us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0808 22:22:13.685415 (+ 19356us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34275/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0808 22:22:13.685621 (+ 206us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43162: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34275/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":52}
W20260808 22:22:13.685794 17843 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43162: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34275/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260808 22:22:13.685904 17882 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:22:13.662873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:43767 (local address 127.0.0.1:43162)
0808 22:22:13.663168 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:13.663183 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:13.663295 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:13.663546 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:13.663549 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:13.663555 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0808 22:22:13.663664 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:13.663671 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:13.664381 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:13.664384 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:13.665391 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:13.665402 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:13.665797 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:13.665801 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:13.665892 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:13.665919 (+ 27us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0808 22:22:13.685578 (+ 19659us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34275/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0808 22:22:13.685737 (+ 159us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.9.176.254:43767: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34275/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":95,"threads_started":1}
I20260808 22:22:13.686465 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 17738
2026-08-08T22:22:13Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (326 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260808 22:22:13.733948 9923 mini_oidc.cc:152] Starting JWKS server
I20260808 22:22:13.734792 9923 webserver.cc:460] Webserver started at https://127.0.0.1:45175/ using document root <none> and password file <none>
I20260808 22:22:13.734879 9923 mini_oidc.cc:189] Starting OIDC Discovery server
I20260808 22:22:13.735098 9923 webserver.cc:460] Webserver started at http://127.0.0.1:40781/ using document root <none> and password file <none>
2026-08-08T22:22:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:22:13Z Disabled control of system clock
I20260808 22:22:13.744325 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786227706282457-9923-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786227706282457-9923-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786227706282457-9923-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:44333
--webserver_interface=127.9.176.254
--webserver_port=0
--builtin_ntp_servers=127.9.176.212:44951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.176.254:44333
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45175/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260808 22:22:13.851226 17892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:22:13.851464 17892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:22:13.851516 17892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:22:13.855482 17892 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:22:13.855536 17892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:22:13.855558 17892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:22:13.855580 17892 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:22:13.855604 17892 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260808 22:22:13.855633 17892 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260808 22:22:13.855654 17892 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45175/jwks.json
I20260808 22:22:13.860641 17892 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:44951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786227706282457-9923-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786227706282457-9923-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.176.254:44333
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:44333
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45175/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.176.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17892
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786227706282457-9923-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:22:13.861640 17892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:22:13.862957 17892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:22:13.871147 17902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:13.871317 17899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:13.871155 17904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:22:13.871411 17892 server_base.cc:1034] running on GCE node
I20260808 22:22:13.872002 17892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:22:13.873035 17892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:22:13.874184 17892 hybrid_clock.cc:648] HybridClock initialized: now 1786227733874164 us; error 36 us; skew 500 ppm
I20260808 22:22:13.875882 17892 webserver.cc:460] Webserver started at http://127.9.176.254:34363/ using document root <none> and password file <none>
I20260808 22:22:13.876406 17892 fs_manager.cc:359] Metadata directory not provided
I20260808 22:22:13.876473 17892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:22:13.876674 17892 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:22:13.878715 17892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786227706282457-9923-0/minicluster-data/master-0/data/instance:
uuid: "c589ebd8fed74bf7bb6b66775fa7814c"
format_stamp: "Formatted at 2026-08-08 22:22:13 on dist-test-slave-ncp9"
server_key: "7aa5c78daa337adbfd7e0df7c308b5b4"
server_key_iv: "02b9c7cfec65638166dfa6935d661703"
server_key_version: "encryptionkey@0"
I20260808 22:22:13.879259 17892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786227706282457-9923-0/minicluster-data/master-0/wal/instance:
uuid: "c589ebd8fed74bf7bb6b66775fa7814c"
format_stamp: "Formatted at 2026-08-08 22:22:13 on dist-test-slave-ncp9"
server_key: "7aa5c78daa337adbfd7e0df7c308b5b4"
server_key_iv: "02b9c7cfec65638166dfa6935d661703"
server_key_version: "encryptionkey@0"
I20260808 22:22:13.882702 17892 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 22:22:13.884814 17912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:13.885963 17892 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 22:22:13.886072 17892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786227706282457-9923-0/minicluster-data/master-0/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786227706282457-9923-0/minicluster-data/master-0/wal
uuid: "c589ebd8fed74bf7bb6b66775fa7814c"
format_stamp: "Formatted at 2026-08-08 22:22:13 on dist-test-slave-ncp9"
server_key: "7aa5c78daa337adbfd7e0df7c308b5b4"
server_key_iv: "02b9c7cfec65638166dfa6935d661703"
server_key_version: "encryptionkey@0"
I20260808 22:22:13.886173 17892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786227706282457-9923-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786227706282457-9923-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786227706282457-9923-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:22:13.903203 17892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:22:13.903780 17892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:22:13.903939 17892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:22:13.910362 17892 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.254:44333
I20260808 22:22:13.910380 17992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.254:44333 every 8 connection(s)
I20260808 22:22:13.911227 17892 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
I20260808 22:22:13.913857 17993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:22:13.919099 17993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c589ebd8fed74bf7bb6b66775fa7814c: Bootstrap starting.
I20260808 22:22:13.919272 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 17892
I20260808 22:22:13.919378 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786227706282457-9923-0/minicluster-data/master-0/wal/instance
I20260808 22:22:13.919620 9923 external_mini_cluster.cc:1428] Setting key 508feda7801950f1d75427dde9229f9e
I20260808 22:22:13.921609 17993 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c589ebd8fed74bf7bb6b66775fa7814c: Neither blocks nor log segments found. Creating new log.
I20260808 22:22:13.922325 17993 log.cc:824] T 00000000000000000000000000000000 P c589ebd8fed74bf7bb6b66775fa7814c: Log is configured to *not* fsync() on all Append() calls
I20260808 22:22:13.924063 17993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c589ebd8fed74bf7bb6b66775fa7814c: No bootstrap required, opened a new log
I20260808 22:22:13.927625 17993 raft_consensus.cc:348] T 00000000000000000000000000000000 P c589ebd8fed74bf7bb6b66775fa7814c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c589ebd8fed74bf7bb6b66775fa7814c" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 44333 } }
I20260808 22:22:13.927841 17993 raft_consensus.cc:374] T 00000000000000000000000000000000 P c589ebd8fed74bf7bb6b66775fa7814c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:22:13.927891 17993 raft_consensus.cc:729] T 00000000000000000000000000000000 P c589ebd8fed74bf7bb6b66775fa7814c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c589ebd8fed74bf7bb6b66775fa7814c, State: Initialized, Role: FOLLOWER
I20260808 22:22:13.928264 17996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:13.920814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40826 (local address 127.9.176.254:44333)
0808 22:22:13.921444 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:13.921461 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:13.921494 (+ 33us) server_negotiation.cc:407] Connection header received
0808 22:22:13.922115 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:13.922132 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:13.922548 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:13.922852 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:13.923254 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:13.924213 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:13.925125 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:13.925605 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:13.925763 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:13.926552 (+ 789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:13.926557 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:13.926569 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:22:13.926932 (+ 363us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:13.926985 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:22:13.927005 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:13.927089 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:13.927229 (+ 140us) server_negotiation.cc:299] Negotiation successful
0808 22:22:13.927465 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":138,"threads_started":1}
I20260808 22:22:13.928287 17993 consensus_queue.cc:260] T 00000000000000000000000000000000 P c589ebd8fed74bf7bb6b66775fa7814c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c589ebd8fed74bf7bb6b66775fa7814c" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 44333 } }
I20260808 22:22:13.928412 17993 raft_consensus.cc:388] T 00000000000000000000000000000000 P c589ebd8fed74bf7bb6b66775fa7814c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:22:13.928471 17993 raft_consensus.cc:482] T 00000000000000000000000000000000 P c589ebd8fed74bf7bb6b66775fa7814c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:22:13.928544 17993 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c589ebd8fed74bf7bb6b66775fa7814c [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:22:13.929430 17993 raft_consensus.cc:504] T 00000000000000000000000000000000 P c589ebd8fed74bf7bb6b66775fa7814c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c589ebd8fed74bf7bb6b66775fa7814c" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 44333 } }
I20260808 22:22:13.929771 17993 leader_election.cc:302] T 00000000000000000000000000000000 P c589ebd8fed74bf7bb6b66775fa7814c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c589ebd8fed74bf7bb6b66775fa7814c; no voters:
I20260808 22:22:13.930078 17993 leader_election.cc:288] T 00000000000000000000000000000000 P c589ebd8fed74bf7bb6b66775fa7814c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:22:13.930177 18004 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c589ebd8fed74bf7bb6b66775fa7814c [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:22:13.930398 18004 raft_consensus.cc:686] T 00000000000000000000000000000000 P c589ebd8fed74bf7bb6b66775fa7814c [term 1 LEADER]: Becoming Leader. State: Replica: c589ebd8fed74bf7bb6b66775fa7814c, State: Running, Role: LEADER
I20260808 22:22:13.930701 18004 consensus_queue.cc:237] T 00000000000000000000000000000000 P c589ebd8fed74bf7bb6b66775fa7814c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c589ebd8fed74bf7bb6b66775fa7814c" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 44333 } }
I20260808 22:22:13.931059 9923 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 22:22:13.931327 17993 sys_catalog.cc:564] T 00000000000000000000000000000000 P c589ebd8fed74bf7bb6b66775fa7814c [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:22:13.932252 18006 sys_catalog.cc:455] T 00000000000000000000000000000000 P c589ebd8fed74bf7bb6b66775fa7814c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c589ebd8fed74bf7bb6b66775fa7814c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c589ebd8fed74bf7bb6b66775fa7814c" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 44333 } } }
I20260808 22:22:13.932374 18006 sys_catalog.cc:458] T 00000000000000000000000000000000 P c589ebd8fed74bf7bb6b66775fa7814c [sys.catalog]: This master's current role is: LEADER
I20260808 22:22:13.932587 18006 sys_catalog.cc:455] T 00000000000000000000000000000000 P c589ebd8fed74bf7bb6b66775fa7814c [sys.catalog]: SysCatalogTable state changed. Reason: New leader c589ebd8fed74bf7bb6b66775fa7814c. Latest consensus state: current_term: 1 leader_uuid: "c589ebd8fed74bf7bb6b66775fa7814c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c589ebd8fed74bf7bb6b66775fa7814c" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 44333 } } }
I20260808 22:22:13.932672 18006 sys_catalog.cc:458] T 00000000000000000000000000000000 P c589ebd8fed74bf7bb6b66775fa7814c [sys.catalog]: This master's current role is: LEADER
I20260808 22:22:13.932983 18022 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:22:13.935745 18022 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:22:13.940514 18022 catalog_manager.cc:1347] Generated new cluster ID: f2f0dbdd8a2c4de18c186620b388890e
I20260808 22:22:13.940577 18022 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:22:13.950548 18022 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:22:13.951339 18022 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:22:13.956058 18022 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c589ebd8fed74bf7bb6b66775fa7814c: Generated new TSK 0
I20260808 22:22:13.956591 18022 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 22:22:13.970674 17996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:13.966254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40830 (local address 127.9.176.254:44333)
0808 22:22:13.966375 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:13.966380 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:13.966549 (+ 169us) server_negotiation.cc:407] Connection header received
0808 22:22:13.966713 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:13.966715 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:13.966759 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:13.966843 (+ 84us) server_negotiation.cc:226] Negotiated authn=JWT
0808 22:22:13.967086 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:13.967589 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:13.968460 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:13.968687 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:13.968774 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:13.969087 (+ 313us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0808 22:22:13.970298 (+ 1211us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45175/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0808 22:22:13.970530 (+ 232us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40830: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45175/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":45}
W20260808 22:22:13.970731 17996 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40830: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45175/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260808 22:22:13.972510 18046 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:22:13.966175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:44333 (local address 127.0.0.1:40830)
0808 22:22:13.966472 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:13.966482 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:13.966581 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:13.966860 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:13.966864 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:13.966871 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0808 22:22:13.966988 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:13.966994 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:13.967718 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:13.967722 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:13.968350 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:13.968358 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:13.968794 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:13.968798 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:13.968882 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:13.968936 (+ 54us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0808 22:22:13.970474 (+ 1538us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45175/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0808 22:22:13.972340 (+ 1866us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.9.176.254:44333: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45175/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":235,"mutex_wait_us":1668,"thread_start_us":89,"threads_started":1}
I20260808 22:22:13.972959 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 17892
E20260808 22:22:13.990819 18047 webserver.cc:533] Webserver: sslize failed:
E20260808 22:22:13.990888 18047 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-08-08T22:22:14Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (287 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-08T22:22:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:22:14Z Disabled control of system clock
I20260808 22:22:14.030591 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786227706282457-9923-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786227706282457-9923-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786227706282457-9923-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:43501
--webserver_interface=127.9.176.254
--webserver_port=0
--builtin_ntp_servers=127.9.176.212:36921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.176.254:43501
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786227706282457-9923-0/testchainca.cert with env {}
W20260808 22:22:14.138669 18055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:22:14.138913 18055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:22:14.138950 18055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:22:14.142800 18055 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:22:14.142846 18055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:22:14.142860 18055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:22:14.142879 18055 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:22:14.142896 18055 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260808 22:22:14.142910 18055 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260808 22:22:14.142927 18055 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260808 22:22:14.147970 18055 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:36921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786227706282457-9923-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786227706282457-9923-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.176.254:43501
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:43501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.176.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786227706282457-9923-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18055
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786227706282457-9923-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:22:14.148859 18055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:22:14.149834 18055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:22:14.157693 18061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:14.157716 18063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:22:14.157802 18055 server_base.cc:1034] running on GCE node
W20260808 22:22:14.157735 18065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:22:14.158524 18055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:22:14.159502 18055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:22:14.160660 18055 hybrid_clock.cc:648] HybridClock initialized: now 1786227734160641 us; error 30 us; skew 500 ppm
I20260808 22:22:14.162388 18055 webserver.cc:460] Webserver started at http://127.9.176.254:42771/ using document root <none> and password file <none>
I20260808 22:22:14.162906 18055 fs_manager.cc:359] Metadata directory not provided
I20260808 22:22:14.162967 18055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:22:14.163157 18055 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:22:14.165166 18055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786227706282457-9923-0/minicluster-data/master-0/data/instance:
uuid: "894837f9f85a4016a68124878d3ecdd4"
format_stamp: "Formatted at 2026-08-08 22:22:14 on dist-test-slave-ncp9"
server_key: "0a9805dc147971273662445bc73ede78"
server_key_iv: "d56a19e7a5c5ecefc227d9c34ec429bf"
server_key_version: "encryptionkey@0"
I20260808 22:22:14.165709 18055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786227706282457-9923-0/minicluster-data/master-0/wal/instance:
uuid: "894837f9f85a4016a68124878d3ecdd4"
format_stamp: "Formatted at 2026-08-08 22:22:14 on dist-test-slave-ncp9"
server_key: "0a9805dc147971273662445bc73ede78"
server_key_iv: "d56a19e7a5c5ecefc227d9c34ec429bf"
server_key_version: "encryptionkey@0"
I20260808 22:22:14.169124 18055 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 22:22:14.171315 18075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:14.172429 18055 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 22:22:14.172539 18055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786227706282457-9923-0/minicluster-data/master-0/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786227706282457-9923-0/minicluster-data/master-0/wal
uuid: "894837f9f85a4016a68124878d3ecdd4"
format_stamp: "Formatted at 2026-08-08 22:22:14 on dist-test-slave-ncp9"
server_key: "0a9805dc147971273662445bc73ede78"
server_key_iv: "d56a19e7a5c5ecefc227d9c34ec429bf"
server_key_version: "encryptionkey@0"
I20260808 22:22:14.172631 18055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786227706282457-9923-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786227706282457-9923-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786227706282457-9923-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:22:14.182217 18055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:22:14.182804 18055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:22:14.182966 18055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:22:14.189378 18055 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.254:43501
I20260808 22:22:14.189386 18144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.254:43501 every 8 connection(s)
I20260808 22:22:14.190249 18055 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
I20260808 22:22:14.193289 18146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:22:14.195372 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 18055
I20260808 22:22:14.195485 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786227706282457-9923-0/minicluster-data/master-0/wal/instance
I20260808 22:22:14.195736 9923 external_mini_cluster.cc:1428] Setting key 20b22ff63e535b0d1c486e71ed14f452
I20260808 22:22:14.198859 18146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 894837f9f85a4016a68124878d3ecdd4: Bootstrap starting.
I20260808 22:22:14.201395 18146 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 894837f9f85a4016a68124878d3ecdd4: Neither blocks nor log segments found. Creating new log.
I20260808 22:22:14.202124 18146 log.cc:824] T 00000000000000000000000000000000 P 894837f9f85a4016a68124878d3ecdd4: Log is configured to *not* fsync() on all Append() calls
I20260808 22:22:14.204126 18146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 894837f9f85a4016a68124878d3ecdd4: No bootstrap required, opened a new log
I20260808 22:22:14.204617 18151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:14.197047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46076 (local address 127.9.176.254:43501)
0808 22:22:14.197619 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:14.197635 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:14.197677 (+ 42us) server_negotiation.cc:407] Connection header received
0808 22:22:14.198281 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:14.198297 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:14.198684 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:14.198961 (+ 277us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:14.199328 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:14.200274 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:14.201492 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:14.201959 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:14.202076 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:14.202943 (+ 867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:14.202947 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:14.202959 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:22:14.203321 (+ 362us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:14.203374 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:22:14.203394 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:14.203468 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:14.203620 (+ 152us) server_negotiation.cc:299] Negotiation successful
0808 22:22:14.203882 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":111,"threads_started":1}
I20260808 22:22:14.207409 9923 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 22:22:14.208062 18146 raft_consensus.cc:348] T 00000000000000000000000000000000 P 894837f9f85a4016a68124878d3ecdd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "894837f9f85a4016a68124878d3ecdd4" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 43501 } }
I20260808 22:22:14.208338 18146 raft_consensus.cc:374] T 00000000000000000000000000000000 P 894837f9f85a4016a68124878d3ecdd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:22:14.208400 18146 raft_consensus.cc:729] T 00000000000000000000000000000000 P 894837f9f85a4016a68124878d3ecdd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 894837f9f85a4016a68124878d3ecdd4, State: Initialized, Role: FOLLOWER
I20260808 22:22:14.208844 18146 consensus_queue.cc:260] T 00000000000000000000000000000000 P 894837f9f85a4016a68124878d3ecdd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "894837f9f85a4016a68124878d3ecdd4" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 43501 } }
I20260808 22:22:14.209023 18146 raft_consensus.cc:388] T 00000000000000000000000000000000 P 894837f9f85a4016a68124878d3ecdd4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:22:14.209093 18146 raft_consensus.cc:482] T 00000000000000000000000000000000 P 894837f9f85a4016a68124878d3ecdd4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:22:14.209196 18146 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 894837f9f85a4016a68124878d3ecdd4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:22:14.209992 18146 raft_consensus.cc:504] T 00000000000000000000000000000000 P 894837f9f85a4016a68124878d3ecdd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "894837f9f85a4016a68124878d3ecdd4" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 43501 } }
I20260808 22:22:14.210311 18146 leader_election.cc:302] T 00000000000000000000000000000000 P 894837f9f85a4016a68124878d3ecdd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 894837f9f85a4016a68124878d3ecdd4; no voters:
I20260808 22:22:14.210592 18146 leader_election.cc:288] T 00000000000000000000000000000000 P 894837f9f85a4016a68124878d3ecdd4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:22:14.210851 18155 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 894837f9f85a4016a68124878d3ecdd4 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:22:14.211090 18155 raft_consensus.cc:686] T 00000000000000000000000000000000 P 894837f9f85a4016a68124878d3ecdd4 [term 1 LEADER]: Becoming Leader. State: Replica: 894837f9f85a4016a68124878d3ecdd4, State: Running, Role: LEADER
I20260808 22:22:14.211426 18155 consensus_queue.cc:237] T 00000000000000000000000000000000 P 894837f9f85a4016a68124878d3ecdd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "894837f9f85a4016a68124878d3ecdd4" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 43501 } }
I20260808 22:22:14.211753 18146 sys_catalog.cc:564] T 00000000000000000000000000000000 P 894837f9f85a4016a68124878d3ecdd4 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:22:14.212831 18159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 894837f9f85a4016a68124878d3ecdd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 894837f9f85a4016a68124878d3ecdd4. Latest consensus state: current_term: 1 leader_uuid: "894837f9f85a4016a68124878d3ecdd4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "894837f9f85a4016a68124878d3ecdd4" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 43501 } } }
I20260808 22:22:14.212972 18159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 894837f9f85a4016a68124878d3ecdd4 [sys.catalog]: This master's current role is: LEADER
I20260808 22:22:14.213025 18156 sys_catalog.cc:455] T 00000000000000000000000000000000 P 894837f9f85a4016a68124878d3ecdd4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "894837f9f85a4016a68124878d3ecdd4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "894837f9f85a4016a68124878d3ecdd4" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 43501 } } }
I20260808 22:22:14.213094 18156 sys_catalog.cc:458] T 00000000000000000000000000000000 P 894837f9f85a4016a68124878d3ecdd4 [sys.catalog]: This master's current role is: LEADER
W20260808 22:22:14.216446 18178 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 894837f9f85a4016a68124878d3ecdd4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 22:22:14.216513 18178 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 22:22:14.216584 18168 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:22:14.217319 18168 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:22:14.222108 18168 catalog_manager.cc:1347] Generated new cluster ID: 6439fcdd1f924b57a2755ce9831dee4d
I20260808 22:22:14.222167 18168 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:22:14.235301 18168 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:22:14.236523 18168 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:22:14.252202 18168 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 894837f9f85a4016a68124878d3ecdd4: Generated new TSK 0
I20260808 22:22:14.253077 18168 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 22:22:14.312282 18151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:14.266059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46078 (local address 127.9.176.254:43501)
0808 22:22:14.266180 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:14.266184 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:14.266337 (+ 153us) server_negotiation.cc:407] Connection header received
0808 22:22:14.266468 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:14.266471 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:14.266526 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:14.266638 (+ 112us) server_negotiation.cc:226] Negotiated authn=JWT
0808 22:22:14.266822 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:14.267670 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:14.268480 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:14.268829 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:14.268953 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:14.269201 (+ 248us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0808 22:22:14.311816 (+ 42615us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0808 22:22:14.312116 (+ 300us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46078: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":33}
W20260808 22:22:14.312376 18151 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46078: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260808 22:22:14.312391 18194 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:22:14.265941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:43501 (local address 127.0.0.1:46078)
0808 22:22:14.266261 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:14.266273 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:14.266365 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:14.266628 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:14.266630 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:14.266636 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0808 22:22:14.266736 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:14.266740 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:14.267758 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:14.267760 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:14.268384 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:14.268393 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:14.268950 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:14.268952 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:14.269022 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:14.269044 (+ 22us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0808 22:22:14.312041 (+ 42997us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0808 22:22:14.312228 (+ 187us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.9.176.254:43501: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":109,"threads_started":1}
I20260808 22:22:14.313385 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 18055
2026-08-08T22:22:14Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (320 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (122 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (122 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 22:22:14 dist-test-slave-ncp9 krb5kdc[18213](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 22:22:14 dist-test-slave-ncp9 krb5kdc[18213](info): set up 2 sockets
Aug 08 22:22:14 dist-test-slave-ncp9 krb5kdc[18213](info): commencing operation
krb5kdc: starting...
W20260808 22:22:15.685643 9923 mini_kdc.cc:121] Time spent starting KDC: real 1.099s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/krb5kdc/test-user.keytab.
Aug 08 22:22:15 dist-test-slave-ncp9 krb5kdc[18213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227735, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T22:22:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:22:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.176.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.176.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.176.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab.
Entry for principal kudu/127.9.176.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab.
I20260808 22:22:15.801465 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:36803
--webserver_interface=127.9.176.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab
--principal=kudu/127.9.176.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:36351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.176.254:36803
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:22:15.909462 18234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:22:15.909685 18234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:22:15.909720 18234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:22:15.913730 18234 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:22:15.913787 18234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:22:15.913805 18234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:22:15.913820 18234 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:22:15.913832 18234 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:22:15.919046 18234 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:36351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.176.254:36803
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab
--principal=kudu/127.9.176.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:36803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.176.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18234
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:22:15.919932 18234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:22:15.920825 18234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:22:15.928911 18247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:22:15.929004 18234 server_base.cc:1034] running on GCE node
W20260808 22:22:15.929073 18242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:15.928929 18241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:22:15.929737 18234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:22:15.930733 18234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:22:15.931900 18234 hybrid_clock.cc:648] HybridClock initialized: now 1786227735931882 us; error 31 us; skew 500 ppm
Aug 08 22:22:15 dist-test-slave-ncp9 krb5kdc[18213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227735, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:22:15.934571 18234 init.cc:377] Logged in from keytab as kudu/127.9.176.254@KRBTEST.COM (short username kudu)
I20260808 22:22:15.935369 18234 webserver.cc:460] Webserver started at http://127.9.176.254:37157/ using document root <none> and password file <none>
I20260808 22:22:15.935870 18234 fs_manager.cc:359] Metadata directory not provided
I20260808 22:22:15.935936 18234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:22:15.936131 18234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:22:15.938060 18234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/minicluster-data/master-0/data/instance:
uuid: "8507fdbf8abf45509418c9aa192d5650"
format_stamp: "Formatted at 2026-08-08 22:22:15 on dist-test-slave-ncp9"
server_key: "97bdd613c079f66e3e8ccc6353c49243"
server_key_iv: "8ad9eed62990d90ced8df06dd79c7a27"
server_key_version: "encryptionkey@0"
I20260808 22:22:15.938593 18234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/minicluster-data/master-0/wal/instance:
uuid: "8507fdbf8abf45509418c9aa192d5650"
format_stamp: "Formatted at 2026-08-08 22:22:15 on dist-test-slave-ncp9"
server_key: "97bdd613c079f66e3e8ccc6353c49243"
server_key_iv: "8ad9eed62990d90ced8df06dd79c7a27"
server_key_version: "encryptionkey@0"
I20260808 22:22:15.942014 18234 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260808 22:22:15.944108 18256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:15.945246 18234 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 22:22:15.945366 18234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/minicluster-data/master-0/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/minicluster-data/master-0/wal
uuid: "8507fdbf8abf45509418c9aa192d5650"
format_stamp: "Formatted at 2026-08-08 22:22:15 on dist-test-slave-ncp9"
server_key: "97bdd613c079f66e3e8ccc6353c49243"
server_key_iv: "8ad9eed62990d90ced8df06dd79c7a27"
server_key_version: "encryptionkey@0"
I20260808 22:22:15.945452 18234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:22:15.960790 18234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:22:15.963269 18234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:22:15.963433 18234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:22:15.969787 18234 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.254:36803
I20260808 22:22:15.969816 18333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.254:36803 every 8 connection(s)
I20260808 22:22:15.970679 18234 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
I20260808 22:22:15.973244 18338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:22:15.976871 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 18234
I20260808 22:22:15.977034 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/minicluster-data/master-0/wal/instance
I20260808 22:22:15.977345 9923 external_mini_cluster.cc:1428] Setting key bd97fc39ea53dc4414a6e64979eeb869
I20260808 22:22:15.979118 18338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8507fdbf8abf45509418c9aa192d5650: Bootstrap starting.
I20260808 22:22:15.981213 18338 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8507fdbf8abf45509418c9aa192d5650: Neither blocks nor log segments found. Creating new log.
I20260808 22:22:15.981894 18338 log.cc:824] T 00000000000000000000000000000000 P 8507fdbf8abf45509418c9aa192d5650: Log is configured to *not* fsync() on all Append() calls
I20260808 22:22:15.983888 18338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8507fdbf8abf45509418c9aa192d5650: No bootstrap required, opened a new log
Aug 08 22:22:15 dist-test-slave-ncp9 krb5kdc[18213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227735, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
I20260808 22:22:15.988061 18338 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8507fdbf8abf45509418c9aa192d5650 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8507fdbf8abf45509418c9aa192d5650" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 36803 } }
I20260808 22:22:15.988317 18338 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8507fdbf8abf45509418c9aa192d5650 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:22:15.988384 18338 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8507fdbf8abf45509418c9aa192d5650 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8507fdbf8abf45509418c9aa192d5650, State: Initialized, Role: FOLLOWER
I20260808 22:22:15.988842 18338 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8507fdbf8abf45509418c9aa192d5650 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8507fdbf8abf45509418c9aa192d5650" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 36803 } }
I20260808 22:22:15.989006 18338 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8507fdbf8abf45509418c9aa192d5650 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:22:15.989071 18338 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8507fdbf8abf45509418c9aa192d5650 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:22:15.989166 18338 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8507fdbf8abf45509418c9aa192d5650 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:22:15.990049 18338 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8507fdbf8abf45509418c9aa192d5650 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8507fdbf8abf45509418c9aa192d5650" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 36803 } }
I20260808 22:22:15.990409 18338 leader_election.cc:302] T 00000000000000000000000000000000 P 8507fdbf8abf45509418c9aa192d5650 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8507fdbf8abf45509418c9aa192d5650; no voters:
I20260808 22:22:15.990724 18338 leader_election.cc:288] T 00000000000000000000000000000000 P 8507fdbf8abf45509418c9aa192d5650 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:22:15.990818 18346 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8507fdbf8abf45509418c9aa192d5650 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:22:15.991039 18346 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8507fdbf8abf45509418c9aa192d5650 [term 1 LEADER]: Becoming Leader. State: Replica: 8507fdbf8abf45509418c9aa192d5650, State: Running, Role: LEADER
I20260808 22:22:15.991338 18346 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8507fdbf8abf45509418c9aa192d5650 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8507fdbf8abf45509418c9aa192d5650" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 36803 } }
I20260808 22:22:15.991581 18343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:15.978723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56868 (local address 127.9.176.254:36803)
0808 22:22:15.979189 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:15.979203 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:15.979242 (+ 39us) server_negotiation.cc:407] Connection header received
0808 22:22:15.979862 (+ 620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:15.979878 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:15.980183 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:15.980467 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:15.981322 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:15.982391 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:15.983524 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:15.983971 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:15.984124 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:15.986760 (+ 2636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:15.986768 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:15.986786 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:22:15.986819 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:15.988627 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:15.989099 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:15.989105 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:15.989111 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:15.989244 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:15.989522 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:15.989524 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:15.989525 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:15.989871 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:15.990017 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:15.990356 (+ 339us) server_negotiation.cc:299] Negotiation successful
0808 22:22:15.990653 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":131,"threads_started":1}
I20260808 22:22:15.991891 18338 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8507fdbf8abf45509418c9aa192d5650 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:22:15.992501 18346 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8507fdbf8abf45509418c9aa192d5650 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8507fdbf8abf45509418c9aa192d5650. Latest consensus state: current_term: 1 leader_uuid: "8507fdbf8abf45509418c9aa192d5650" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8507fdbf8abf45509418c9aa192d5650" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 36803 } } }
I20260808 22:22:15.992612 18346 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8507fdbf8abf45509418c9aa192d5650 [sys.catalog]: This master's current role is: LEADER
I20260808 22:22:15.992914 18350 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8507fdbf8abf45509418c9aa192d5650 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8507fdbf8abf45509418c9aa192d5650" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8507fdbf8abf45509418c9aa192d5650" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 36803 } } }
I20260808 22:22:15.993014 18350 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8507fdbf8abf45509418c9aa192d5650 [sys.catalog]: This master's current role is: LEADER
I20260808 22:22:15.994032 18355 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:22:15.996796 18355 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:22:16.002018 18355 catalog_manager.cc:1347] Generated new cluster ID: a9006a3a8a1442069c207a36436a2d67
I20260808 22:22:16.002067 18355 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:22:16.017840 18355 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:22:16.019088 18355 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:22:16.024766 18355 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8507fdbf8abf45509418c9aa192d5650: Generated new TSK 0
I20260808 22:22:16.025547 18355 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.176.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.176.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.176.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.193.keytab.
Entry for principal kudu/127.9.176.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.193.keytab.
I20260808 22:22:16.053885 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.193:0
--local_ip_for_outbound_sockets=127.9.176.193
--webserver_interface=127.9.176.193
--webserver_port=0
--tserver_master_addrs=127.9.176.254:36803
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.193.keytab
--principal=kudu/127.9.176.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:36351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:22:16.163527 18377 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:22:16.163775 18377 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:22:16.163841 18377 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:22:16.167771 18377 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:22:16.167831 18377 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:22:16.167920 18377 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.193
I20260808 22:22:16.172868 18377 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:36351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.193.keytab
--principal=kudu/127.9.176.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.176.193
--webserver_port=0
--tserver_master_addrs=127.9.176.254:36803
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18377
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.193
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:22:16.173851 18377 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:22:16.174822 18377 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:22:16.184283 18389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:16.184316 18390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:16.184324 18392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:22:16.184389 18377 server_base.cc:1034] running on GCE node
I20260808 22:22:16.184959 18377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:22:16.185593 18377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:22:16.186749 18377 hybrid_clock.cc:648] HybridClock initialized: now 1786227736186734 us; error 31 us; skew 500 ppm
Aug 08 22:22:16 dist-test-slave-ncp9 krb5kdc[18213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227736, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:22:16.189450 18377 init.cc:377] Logged in from keytab as kudu/127.9.176.193@KRBTEST.COM (short username kudu)
I20260808 22:22:16.190269 18377 webserver.cc:460] Webserver started at http://127.9.176.193:32781/ using document root <none> and password file <none>
I20260808 22:22:16.190770 18377 fs_manager.cc:359] Metadata directory not provided
I20260808 22:22:16.190827 18377 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:22:16.190994 18377 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:22:16.193059 18377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/minicluster-data/ts-0/data/instance:
uuid: "754697307e834dfbbb322d0068910ebe"
format_stamp: "Formatted at 2026-08-08 22:22:16 on dist-test-slave-ncp9"
server_key: "7ad9904cffcc5867e783c564b48bffdf"
server_key_iv: "6205e39c6a03d92cf4f35ec81d16dc6a"
server_key_version: "encryptionkey@0"
I20260808 22:22:16.193579 18377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/minicluster-data/ts-0/wal/instance:
uuid: "754697307e834dfbbb322d0068910ebe"
format_stamp: "Formatted at 2026-08-08 22:22:16 on dist-test-slave-ncp9"
server_key: "7ad9904cffcc5867e783c564b48bffdf"
server_key_iv: "6205e39c6a03d92cf4f35ec81d16dc6a"
server_key_version: "encryptionkey@0"
I20260808 22:22:16.197074 18377 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260808 22:22:16.199244 18401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:16.200331 18377 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 22:22:16.200429 18377 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/minicluster-data/ts-0/wal
uuid: "754697307e834dfbbb322d0068910ebe"
format_stamp: "Formatted at 2026-08-08 22:22:16 on dist-test-slave-ncp9"
server_key: "7ad9904cffcc5867e783c564b48bffdf"
server_key_iv: "6205e39c6a03d92cf4f35ec81d16dc6a"
server_key_version: "encryptionkey@0"
I20260808 22:22:16.200527 18377 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:22:16.220427 18377 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:22:16.223358 18377 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:22:16.223528 18377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:22:16.224112 18377 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:22:16.224932 18377 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:22:16.224977 18377 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:16.225029 18377 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:22:16.225059 18377 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:16.233429 18377 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.193:41045
I20260808 22:22:16.233462 18576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.193:41045 every 8 connection(s)
I20260808 22:22:16.234280 18377 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
I20260808 22:22:16.239414 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 18377
I20260808 22:22:16.239521 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/minicluster-data/ts-0/wal/instance
I20260808 22:22:16.239765 9923 external_mini_cluster.cc:1428] Setting key 50f3ba66d5e6724dcda9ef4e9ea1d5f5
Aug 08 22:22:16 dist-test-slave-ncp9 krb5kdc[18213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227736, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.193@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
I20260808 22:22:16.248554 18343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:16.235962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.193:56819 (local address 127.9.176.254:36803)
0808 22:22:16.236103 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:16.236108 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:16.237016 (+ 908us) server_negotiation.cc:407] Connection header received
0808 22:22:16.238378 (+ 1362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:16.238381 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:16.238427 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:16.238542 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:16.239658 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:16.240302 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:16.241646 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:16.241871 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:16.241951 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:16.244876 (+ 2925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:16.244880 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:16.244882 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:22:16.244936 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:16.246474 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:16.246962 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:16.246966 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:16.246967 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:16.247065 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:16.247452 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:16.247455 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:16.247457 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:16.247647 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:16.247736 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:16.248235 (+ 499us) server_negotiation.cc:299] Negotiation successful
0808 22:22:16.248361 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 22:22:16.249353 18579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:16.236368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:36803 (local address 127.9.176.193:56819)
0808 22:22:16.236835 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:16.236879 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:16.238172 (+ 1293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:16.238676 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:16.238683 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:16.239031 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:16.239502 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:16.239522 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:16.240431 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:16.240435 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:16.241529 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:16.241542 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:16.242023 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:16.242027 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:16.242225 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:16.242801 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:16.242819 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:16.244702 (+ 1883us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:22:16.246614 (+ 1912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:16.246619 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:16.246629 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:16.246868 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:16.247183 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:16.247186 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:16.247188 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:16.247326 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:16.247749 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:22:16.247756 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:22:16.247988 (+ 232us) client_negotiation.cc:769] Sending connection context
0808 22:22:16.248183 (+ 195us) client_negotiation.cc:240] Negotiation successful
0808 22:22:16.248488 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":122,"threads_started":1}
WARNING: no policy specified for kudu/127.9.176.194@KRBTEST.COM; defaulting to no policy
I20260808 22:22:16.250820 18577 heartbeater.cc:347] Connected to a master server at 127.9.176.254:36803
I20260808 22:22:16.251034 18577 heartbeater.cc:464] Registering TS with master...
I20260808 22:22:16.251470 18577 heartbeater.cc:511] Master 127.9.176.254:36803 requested a full tablet report, sending...
I20260808 22:22:16.252951 18281 ts_manager.cc:194] Registered new tserver with Master: 754697307e834dfbbb322d0068910ebe (127.9.176.193:41045)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.176.194@KRBTEST.COM" created.
I20260808 22:22:16.254163 18281 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.176.193@KRBTEST.COM'} at 127.9.176.193:56819
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.176.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.194.keytab.
Entry for principal kudu/127.9.176.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.194.keytab.
I20260808 22:22:16.268102 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.194:0
--local_ip_for_outbound_sockets=127.9.176.194
--webserver_interface=127.9.176.194
--webserver_port=0
--tserver_master_addrs=127.9.176.254:36803
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.194.keytab
--principal=kudu/127.9.176.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:36351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:22:16.374308 18583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:22:16.374545 18583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:22:16.374598 18583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:22:16.378481 18583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:22:16.378532 18583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:22:16.378604 18583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.194
I20260808 22:22:16.383730 18583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:36351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.194.keytab
--principal=kudu/127.9.176.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.176.194
--webserver_port=0
--tserver_master_addrs=127.9.176.254:36803
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.194
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:22:16.384618 18583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:22:16.385586 18583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:22:16.394793 18592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:16.394802 18591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:16.394801 18594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:22:16.394943 18583 server_base.cc:1034] running on GCE node
I20260808 22:22:16.395437 18583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:22:16.396025 18583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:22:16.397173 18583 hybrid_clock.cc:648] HybridClock initialized: now 1786227736397151 us; error 42 us; skew 500 ppm
Aug 08 22:22:16 dist-test-slave-ncp9 krb5kdc[18213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227736, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:22:16.399813 18583 init.cc:377] Logged in from keytab as kudu/127.9.176.194@KRBTEST.COM (short username kudu)
I20260808 22:22:16.400601 18583 webserver.cc:460] Webserver started at http://127.9.176.194:44723/ using document root <none> and password file <none>
I20260808 22:22:16.401154 18583 fs_manager.cc:359] Metadata directory not provided
I20260808 22:22:16.401221 18583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:22:16.401418 18583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:22:16.403434 18583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/minicluster-data/ts-1/data/instance:
uuid: "a611401f53074268a3ac2854ff6fe93b"
format_stamp: "Formatted at 2026-08-08 22:22:16 on dist-test-slave-ncp9"
server_key: "94bb2d80d9d893247b88ca96170a6314"
server_key_iv: "fcb486732ea72d34e51056d7586d5b0a"
server_key_version: "encryptionkey@0"
I20260808 22:22:16.404042 18583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/minicluster-data/ts-1/wal/instance:
uuid: "a611401f53074268a3ac2854ff6fe93b"
format_stamp: "Formatted at 2026-08-08 22:22:16 on dist-test-slave-ncp9"
server_key: "94bb2d80d9d893247b88ca96170a6314"
server_key_iv: "fcb486732ea72d34e51056d7586d5b0a"
server_key_version: "encryptionkey@0"
I20260808 22:22:16.407545 18583 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260808 22:22:16.409754 18606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:16.410880 18583 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 22:22:16.410984 18583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/minicluster-data/ts-1/wal
uuid: "a611401f53074268a3ac2854ff6fe93b"
format_stamp: "Formatted at 2026-08-08 22:22:16 on dist-test-slave-ncp9"
server_key: "94bb2d80d9d893247b88ca96170a6314"
server_key_iv: "fcb486732ea72d34e51056d7586d5b0a"
server_key_version: "encryptionkey@0"
I20260808 22:22:16.411070 18583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:22:16.455497 18583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:22:16.458469 18583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:22:16.458643 18583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:22:16.459225 18583 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:22:16.460044 18583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:22:16.460093 18583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:16.460146 18583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:22:16.460179 18583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:16.468357 18583 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.194:36841
I20260808 22:22:16.468410 18772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.194:36841 every 8 connection(s)
I20260808 22:22:16.469267 18583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
I20260808 22:22:16.473855 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 18583
I20260808 22:22:16.473964 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/minicluster-data/ts-1/wal/instance
I20260808 22:22:16.474222 9923 external_mini_cluster.cc:1428] Setting key be9107aaf3f2b90e51a2e0bc3d20493e
Aug 08 22:22:16 dist-test-slave-ncp9 krb5kdc[18213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227736, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.194@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
I20260808 22:22:16.482311 18343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:16.470902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.194:54465 (local address 127.9.176.254:36803)
0808 22:22:16.471030 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:16.471033 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:16.471813 (+ 780us) server_negotiation.cc:407] Connection header received
0808 22:22:16.473166 (+ 1353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:16.473169 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:16.473227 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:16.473298 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:16.474459 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:16.475082 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:16.476279 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:16.476494 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:16.476587 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:16.479106 (+ 2519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:16.479109 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:16.479111 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:22:16.479134 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:16.480389 (+ 1255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:16.480854 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:16.480857 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:16.480858 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:16.480969 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:16.481272 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:16.481276 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:16.481277 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:16.481447 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:16.481532 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:16.482001 (+ 469us) server_negotiation.cc:299] Negotiation successful
0808 22:22:16.482134 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260808 22:22:16.483175 18778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:16.471230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:36803 (local address 127.9.176.194:54465)
0808 22:22:16.471659 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:16.471699 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:16.472944 (+ 1245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:16.473473 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:16.473480 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:16.473813 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:16.474322 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:16.474340 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:16.475202 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:16.475205 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:16.476164 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:16.476175 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:16.476633 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:16.476637 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:16.476794 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:16.477403 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:16.477422 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:16.478952 (+ 1530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:22:16.480492 (+ 1540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:16.480498 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:16.480520 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:16.480761 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:16.481049 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:16.481052 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:16.481054 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:16.481187 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:16.481552 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:22:16.481558 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:22:16.481783 (+ 225us) client_negotiation.cc:769] Sending connection context
0808 22:22:16.481943 (+ 160us) client_negotiation.cc:240] Negotiation successful
0808 22:22:16.482266 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":116,"threads_started":1}
WARNING: no policy specified for kudu/127.9.176.195@KRBTEST.COM; defaulting to no policy
I20260808 22:22:16.484239 18775 heartbeater.cc:347] Connected to a master server at 127.9.176.254:36803
I20260808 22:22:16.484462 18775 heartbeater.cc:464] Registering TS with master...
I20260808 22:22:16.484954 18775 heartbeater.cc:511] Master 127.9.176.254:36803 requested a full tablet report, sending...
I20260808 22:22:16.486013 18281 ts_manager.cc:194] Registered new tserver with Master: a611401f53074268a3ac2854ff6fe93b (127.9.176.194:36841)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.176.195@KRBTEST.COM" created.
I20260808 22:22:16.486696 18281 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.176.194@KRBTEST.COM'} at 127.9.176.194:54465
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.176.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.195.keytab.
Entry for principal kudu/127.9.176.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.195.keytab.
I20260808 22:22:16.501353 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.195:0
--local_ip_for_outbound_sockets=127.9.176.195
--webserver_interface=127.9.176.195
--webserver_port=0
--tserver_master_addrs=127.9.176.254:36803
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.195.keytab
--principal=kudu/127.9.176.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:36351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:22:16.611150 18782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:22:16.611434 18782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:22:16.611485 18782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:22:16.616706 18782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:22:16.616760 18782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:22:16.616815 18782 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.195
I20260808 22:22:16.622030 18782 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:36351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.195.keytab
--principal=kudu/127.9.176.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.176.195
--webserver_port=0
--tserver_master_addrs=127.9.176.254:36803
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18782
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.195
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:22:16.623000 18782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:22:16.623943 18782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:22:16.632970 18789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:16.633023 18790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:16.633016 18793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:22:16.633227 18782 server_base.cc:1034] running on GCE node
I20260808 22:22:16.633579 18782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:22:16.634119 18782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:22:16.635272 18782 hybrid_clock.cc:648] HybridClock initialized: now 1786227736635258 us; error 32 us; skew 500 ppm
Aug 08 22:22:16 dist-test-slave-ncp9 krb5kdc[18213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227736, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:22:16.637832 18782 init.cc:377] Logged in from keytab as kudu/127.9.176.195@KRBTEST.COM (short username kudu)
I20260808 22:22:16.638610 18782 webserver.cc:460] Webserver started at http://127.9.176.195:37383/ using document root <none> and password file <none>
I20260808 22:22:16.639125 18782 fs_manager.cc:359] Metadata directory not provided
I20260808 22:22:16.639194 18782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:22:16.639381 18782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:22:16.641314 18782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/minicluster-data/ts-2/data/instance:
uuid: "12c2ad5ce4d641e0a21bf2c6d65e1191"
format_stamp: "Formatted at 2026-08-08 22:22:16 on dist-test-slave-ncp9"
server_key: "a1a22ba1a399bf6066879132e402a678"
server_key_iv: "67f7547711860bca71fb9fa55a1981e8"
server_key_version: "encryptionkey@0"
I20260808 22:22:16.641826 18782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/minicluster-data/ts-2/wal/instance:
uuid: "12c2ad5ce4d641e0a21bf2c6d65e1191"
format_stamp: "Formatted at 2026-08-08 22:22:16 on dist-test-slave-ncp9"
server_key: "a1a22ba1a399bf6066879132e402a678"
server_key_iv: "67f7547711860bca71fb9fa55a1981e8"
server_key_version: "encryptionkey@0"
I20260808 22:22:16.645254 18782 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260808 22:22:16.647440 18802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:16.648569 18782 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 22:22:16.648669 18782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/minicluster-data/ts-2/wal
uuid: "12c2ad5ce4d641e0a21bf2c6d65e1191"
format_stamp: "Formatted at 2026-08-08 22:22:16 on dist-test-slave-ncp9"
server_key: "a1a22ba1a399bf6066879132e402a678"
server_key_iv: "67f7547711860bca71fb9fa55a1981e8"
server_key_version: "encryptionkey@0"
I20260808 22:22:16.648751 18782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:22:16.659042 18782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:22:16.661476 18782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:22:16.661639 18782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:22:16.662129 18782 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:22:16.662868 18782 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:22:16.662912 18782 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:16.662951 18782 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:22:16.662982 18782 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:16.670792 18782 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.195:34109
I20260808 22:22:16.670842 18977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.195:34109 every 8 connection(s)
I20260808 22:22:16.671576 18782 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
I20260808 22:22:16.677932 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 18782
I20260808 22:22:16.678046 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/minicluster-data/ts-2/wal/instance
I20260808 22:22:16.678345 9923 external_mini_cluster.cc:1428] Setting key 8b88018b89b3954a4cadbb18ce288c52
Aug 08 22:22:16 dist-test-slave-ncp9 krb5kdc[18213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227736, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.195@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
I20260808 22:22:16.686949 18343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:16.673206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.195:59339 (local address 127.9.176.254:36803)
0808 22:22:16.673339 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:16.673342 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:16.674068 (+ 726us) server_negotiation.cc:407] Connection header received
0808 22:22:16.675383 (+ 1315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:16.675386 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:16.675442 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:16.675517 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:16.676614 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:16.677193 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:16.678604 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:16.678904 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:16.679013 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:16.684149 (+ 5136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:16.684152 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:16.684154 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:22:16.684179 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:16.685366 (+ 1187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:16.685795 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:16.685797 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:16.685798 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:16.685871 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:16.686110 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:16.686112 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:16.686113 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:16.686267 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:16.686335 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:16.686739 (+ 404us) server_negotiation.cc:299] Negotiation successful
0808 22:22:16.686835 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 22:22:16.687649 18984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:16.673518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:36803 (local address 127.9.176.195:59339)
0808 22:22:16.673935 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:16.673974 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:16.675181 (+ 1207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:16.675691 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:16.675698 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:16.676016 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:16.676469 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:16.676481 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:16.677319 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:16.677322 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:16.678482 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:16.678494 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:16.679042 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:16.679044 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:16.679215 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:16.682612 (+ 3397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:16.682628 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:16.683981 (+ 1353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:22:16.685455 (+ 1474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:16.685460 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:16.685470 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:16.685715 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:16.685932 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:16.685933 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:16.685935 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:16.686050 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:16.686335 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:22:16.686339 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:22:16.686543 (+ 204us) client_negotiation.cc:769] Sending connection context
0808 22:22:16.686694 (+ 151us) client_negotiation.cc:240] Negotiation successful
0808 22:22:16.686920 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":105,"threads_started":1}
I20260808 22:22:16.688608 18982 heartbeater.cc:347] Connected to a master server at 127.9.176.254:36803
I20260808 22:22:16.688803 18982 heartbeater.cc:464] Registering TS with master...
I20260808 22:22:16.689255 18982 heartbeater.cc:511] Master 127.9.176.254:36803 requested a full tablet report, sending...
I20260808 22:22:16.690184 18281 ts_manager.cc:194] Registered new tserver with Master: 12c2ad5ce4d641e0a21bf2c6d65e1191 (127.9.176.195:34109)
I20260808 22:22:16.690753 18281 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.176.195@KRBTEST.COM'} at 127.9.176.195:59339
I20260808 22:22:16.692071 9923 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/krb5kdc/test-admin.keytab.
Aug 08 22:22:16 dist-test-slave-ncp9 krb5kdc[18213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:22:16.708923 9923 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260808 22:22:16.711448 18993 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:22:16.710198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51714 (local address 127.9.176.193:41045)
0808 22:22:16.710515 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:16.710531 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:16.710558 (+ 27us) server_negotiation.cc:407] Connection header received
0808 22:22:16.710608 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:16.710614 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:16.710795 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:16.710912 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:16.711298 (+ 386us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51714: BlockingRecv error: recv got EOF from 127.0.0.1:51714 (error 108)
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":140,"threads_started":1}
W20260808 22:22:16.711896 9923 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/krb5kdc/krb5cc)
Aug 08 22:22:16 dist-test-slave-ncp9 krb5kdc[18213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:22:16.712833 9923 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 08 22:22:16 dist-test-slave-ncp9 krb5kdc[18213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:22:16.713687 9923 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 08 22:22:16 dist-test-slave-ncp9 krb5kdc[18213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.176.193@KRBTEST.COM
I20260808 22:22:16.723076 18993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:16.714511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51728 (local address 127.9.176.193:41045)
0808 22:22:16.714631 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:16.714635 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:16.714721 (+ 86us) server_negotiation.cc:407] Connection header received
0808 22:22:16.714829 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:16.714831 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:16.714911 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:16.714974 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:16.715501 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:16.716300 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:16.717206 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:16.717829 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:16.717941 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:16.720038 (+ 2097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:16.720049 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:16.720062 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:22:16.720083 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:16.721420 (+ 1337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:16.721818 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:16.721823 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:16.721824 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:16.721901 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:16.722153 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:16.722157 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:16.722159 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:16.722404 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:16.722596 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:16.722805 (+ 209us) server_negotiation.cc:299] Negotiation successful
0808 22:22:16.722951 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260808 22:22:16.726559 18993 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:22:16.725677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51740 (local address 127.9.176.193:41045)
0808 22:22:16.725796 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:16.725799 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:16.725864 (+ 65us) server_negotiation.cc:407] Connection header received
0808 22:22:16.725943 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:16.725945 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:16.725977 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:16.726036 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:16.726467 (+ 431us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51740: BlockingRecv error: recv got EOF from 127.0.0.1:51740 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260808 22:22:16.726822 9923 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/krb5kdc/krb5cc' of type FILE
Aug 08 22:22:16 dist-test-slave-ncp9 krb5kdc[18213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:22:16.727778 9923 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 08 22:22:16 dist-test-slave-ncp9 krb5kdc[18213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:22:16.728645 9923 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 08 22:22:16 dist-test-slave-ncp9 krb5kdc[18213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.176.193@KRBTEST.COM
I20260808 22:22:16.736482 18993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:16.729469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51752 (local address 127.9.176.193:41045)
0808 22:22:16.729578 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:16.729581 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:16.729624 (+ 43us) server_negotiation.cc:407] Connection header received
0808 22:22:16.729716 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:16.729718 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:16.729747 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:16.729803 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:16.730232 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:16.730714 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:16.731450 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:16.731666 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:16.731734 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:16.733786 (+ 2052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:16.733789 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:16.733791 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:22:16.733815 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:16.735203 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:16.735633 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:16.735636 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:16.735637 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:16.735718 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:16.735915 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:16.735917 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:16.735918 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:16.736054 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:16.736139 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:16.736278 (+ 139us) server_negotiation.cc:299] Negotiation successful
0808 22:22:16.736377 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260808 22:22:16.738822 18993 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:22:16.737906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51760 (local address 127.9.176.193:41045)
0808 22:22:16.738032 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:16.738035 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:16.738067 (+ 32us) server_negotiation.cc:407] Connection header received
0808 22:22:16.738159 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:16.738161 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:16.738190 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:16.738246 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:16.738725 (+ 479us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51760: BlockingRecv error: recv got EOF from 127.0.0.1:51760 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260808 22:22:16.738883 9923 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786227706282457-9923-0/krb5kdc/krb5cc' of type FILE
Aug 08 22:22:16 dist-test-slave-ncp9 krb5kdc[18213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:22:16.739799 9923 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 08 22:22:16 dist-test-slave-ncp9 krb5kdc[18213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:22:16.740562 9923 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 08 22:22:16 dist-test-slave-ncp9 krb5kdc[18213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.176.193@KRBTEST.COM
I20260808 22:22:16.748016 18993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:16.741287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51776 (local address 127.9.176.193:41045)
0808 22:22:16.741390 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:16.741394 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:16.741458 (+ 64us) server_negotiation.cc:407] Connection header received
0808 22:22:16.741573 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:16.741575 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:16.741609 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:16.741672 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:16.742113 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:16.742591 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:16.743332 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:16.743535 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:16.743617 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:16.745597 (+ 1980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:16.745600 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:16.745601 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:22:16.745621 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:16.746625 (+ 1004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:16.747003 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:16.747007 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:16.747008 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:16.747078 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:16.747322 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:16.747325 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:16.747327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:16.747468 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:16.747571 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:16.747790 (+ 219us) server_negotiation.cc:299] Negotiation successful
0808 22:22:16.747883 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260808 22:22:16.749022 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 18377
I20260808 22:22:16.762421 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 18583
I20260808 22:22:16.771345 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 18782
I20260808 22:22:16.781198 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 18234
2026-08-08T22:22:16Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2219 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 22:22:16 dist-test-slave-ncp9 krb5kdc[19010](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 22:22:16 dist-test-slave-ncp9 krb5kdc[19010](info): set up 2 sockets
Aug 08 22:22:16 dist-test-slave-ncp9 krb5kdc[19010](info): commencing operation
krb5kdc: starting...
W20260808 22:22:17.900738 9923 mini_kdc.cc:121] Time spent starting KDC: real 1.095s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/krb5kdc/test-user.keytab.
Aug 08 22:22:17 dist-test-slave-ncp9 krb5kdc[19010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227737, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T22:22:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:22:17Z Disabled control of system clock
WARNING: no policy specified for oryx/127.9.176.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.176.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.176.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/krb5kdc/oryx_127.9.176.254.keytab.
Entry for principal oryx/127.9.176.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/krb5kdc/oryx_127.9.176.254.keytab.
I20260808 22:22:18.019469 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:46579
--webserver_interface=127.9.176.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/krb5kdc/oryx_127.9.176.254.keytab
--principal=oryx/127.9.176.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:43621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.176.254:46579
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:22:18.131300 19030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:22:18.131549 19030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:22:18.131598 19030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:22:18.135525 19030 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:22:18.135586 19030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:22:18.135612 19030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:22:18.135648 19030 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:22:18.135677 19030 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:22:18.140868 19030 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:43621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.176.254:46579
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/krb5kdc/oryx_127.9.176.254.keytab
--principal=oryx/127.9.176.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:46579
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.176.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19030
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:22:18.141891 19030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:22:18.142902 19030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:22:18.151163 19040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:18.151146 19041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:22:18.151242 19030 server_base.cc:1034] running on GCE node
W20260808 22:22:18.151147 19043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:22:18.151926 19030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:22:18.153016 19030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:22:18.154202 19030 hybrid_clock.cc:648] HybridClock initialized: now 1786227738154177 us; error 36 us; skew 500 ppm
Aug 08 22:22:18 dist-test-slave-ncp9 krb5kdc[19010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227738, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.176.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:22:18.156930 19030 init.cc:377] Logged in from keytab as oryx/127.9.176.254@KRBTEST.COM (short username oryx)
I20260808 22:22:18.157816 19030 webserver.cc:460] Webserver started at http://127.9.176.254:33787/ using document root <none> and password file <none>
I20260808 22:22:18.158326 19030 fs_manager.cc:359] Metadata directory not provided
I20260808 22:22:18.158392 19030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:22:18.158593 19030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:22:18.160516 19030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/master-0/data/instance:
uuid: "be929da42de644648042b1c514d26d16"
format_stamp: "Formatted at 2026-08-08 22:22:18 on dist-test-slave-ncp9"
server_key: "ab973a9732c319c650f775a921bb316e"
server_key_iv: "5215fb488de1f407e4e6ea12e5ffdeb0"
server_key_version: "encryptionkey@0"
I20260808 22:22:18.161098 19030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/master-0/wal/instance:
uuid: "be929da42de644648042b1c514d26d16"
format_stamp: "Formatted at 2026-08-08 22:22:18 on dist-test-slave-ncp9"
server_key: "ab973a9732c319c650f775a921bb316e"
server_key_iv: "5215fb488de1f407e4e6ea12e5ffdeb0"
server_key_version: "encryptionkey@0"
I20260808 22:22:18.164623 19030 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260808 22:22:18.166913 19051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:18.168046 19030 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 22:22:18.168157 19030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/master-0/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/master-0/wal
uuid: "be929da42de644648042b1c514d26d16"
format_stamp: "Formatted at 2026-08-08 22:22:18 on dist-test-slave-ncp9"
server_key: "ab973a9732c319c650f775a921bb316e"
server_key_iv: "5215fb488de1f407e4e6ea12e5ffdeb0"
server_key_version: "encryptionkey@0"
I20260808 22:22:18.168244 19030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:22:18.191010 19030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:22:18.194051 19030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:22:18.194226 19030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:22:18.200780 19030 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.254:46579
I20260808 22:22:18.201059 19138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.254:46579 every 8 connection(s)
I20260808 22:22:18.201733 19030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
I20260808 22:22:18.204337 19139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:22:18.205713 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 19030
I20260808 22:22:18.205847 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/master-0/wal/instance
I20260808 22:22:18.206156 9923 external_mini_cluster.cc:1428] Setting key 81bd10bd18e933ec7add5f830b911b44
I20260808 22:22:18.210515 19139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be929da42de644648042b1c514d26d16: Bootstrap starting.
I20260808 22:22:18.212996 19139 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be929da42de644648042b1c514d26d16: Neither blocks nor log segments found. Creating new log.
I20260808 22:22:18.213869 19139 log.cc:824] T 00000000000000000000000000000000 P be929da42de644648042b1c514d26d16: Log is configured to *not* fsync() on all Append() calls
Aug 08 22:22:18 dist-test-slave-ncp9 krb5kdc[19010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227737, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM
I20260808 22:22:18.215807 19139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be929da42de644648042b1c514d26d16: No bootstrap required, opened a new log
I20260808 22:22:18.220291 19143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:18.207421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34800 (local address 127.9.176.254:46579)
0808 22:22:18.207801 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:18.207819 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:18.207853 (+ 34us) server_negotiation.cc:407] Connection header received
0808 22:22:18.208456 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:18.208472 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:18.208791 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:18.209083 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:18.209929 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:18.210936 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:18.212027 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:18.212470 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:18.212586 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:18.215403 (+ 2817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:18.215408 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:18.215421 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:22:18.215447 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:18.217326 (+ 1879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:18.217742 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:18.217749 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:18.217755 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:18.217914 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:18.218139 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:18.218142 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:18.218144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:18.218641 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:18.218844 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:18.219021 (+ 177us) server_negotiation.cc:299] Negotiation successful
0808 22:22:18.219325 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":108,"threads_started":1}
I20260808 22:22:18.220170 19139 raft_consensus.cc:348] T 00000000000000000000000000000000 P be929da42de644648042b1c514d26d16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be929da42de644648042b1c514d26d16" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 46579 } }
I20260808 22:22:18.220428 19139 raft_consensus.cc:374] T 00000000000000000000000000000000 P be929da42de644648042b1c514d26d16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:22:18.220485 19139 raft_consensus.cc:729] T 00000000000000000000000000000000 P be929da42de644648042b1c514d26d16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be929da42de644648042b1c514d26d16, State: Initialized, Role: FOLLOWER
I20260808 22:22:18.220979 19139 consensus_queue.cc:260] T 00000000000000000000000000000000 P be929da42de644648042b1c514d26d16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be929da42de644648042b1c514d26d16" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 46579 } }
I20260808 22:22:18.221134 19139 raft_consensus.cc:388] T 00000000000000000000000000000000 P be929da42de644648042b1c514d26d16 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:22:18.221211 19139 raft_consensus.cc:482] T 00000000000000000000000000000000 P be929da42de644648042b1c514d26d16 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:22:18.221308 19139 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be929da42de644648042b1c514d26d16 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:22:18.222175 19139 raft_consensus.cc:504] T 00000000000000000000000000000000 P be929da42de644648042b1c514d26d16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be929da42de644648042b1c514d26d16" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 46579 } }
I20260808 22:22:18.222543 19139 leader_election.cc:302] T 00000000000000000000000000000000 P be929da42de644648042b1c514d26d16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be929da42de644648042b1c514d26d16; no voters:
I20260808 22:22:18.222828 19139 leader_election.cc:288] T 00000000000000000000000000000000 P be929da42de644648042b1c514d26d16 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:22:18.222971 19145 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be929da42de644648042b1c514d26d16 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:22:18.223158 19145 raft_consensus.cc:686] T 00000000000000000000000000000000 P be929da42de644648042b1c514d26d16 [term 1 LEADER]: Becoming Leader. State: Replica: be929da42de644648042b1c514d26d16, State: Running, Role: LEADER
I20260808 22:22:18.223451 19145 consensus_queue.cc:237] T 00000000000000000000000000000000 P be929da42de644648042b1c514d26d16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be929da42de644648042b1c514d26d16" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 46579 } }
I20260808 22:22:18.224031 19139 sys_catalog.cc:564] T 00000000000000000000000000000000 P be929da42de644648042b1c514d26d16 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:22:18.225162 19147 sys_catalog.cc:455] T 00000000000000000000000000000000 P be929da42de644648042b1c514d26d16 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "be929da42de644648042b1c514d26d16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be929da42de644648042b1c514d26d16" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 46579 } } }
I20260808 22:22:18.225272 19147 sys_catalog.cc:458] T 00000000000000000000000000000000 P be929da42de644648042b1c514d26d16 [sys.catalog]: This master's current role is: LEADER
I20260808 22:22:18.225535 19148 sys_catalog.cc:455] T 00000000000000000000000000000000 P be929da42de644648042b1c514d26d16 [sys.catalog]: SysCatalogTable state changed. Reason: New leader be929da42de644648042b1c514d26d16. Latest consensus state: current_term: 1 leader_uuid: "be929da42de644648042b1c514d26d16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be929da42de644648042b1c514d26d16" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 46579 } } }
I20260808 22:22:18.225633 19148 sys_catalog.cc:458] T 00000000000000000000000000000000 P be929da42de644648042b1c514d26d16 [sys.catalog]: This master's current role is: LEADER
I20260808 22:22:18.225711 19156 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:22:18.228484 19156 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:22:18.233724 19156 catalog_manager.cc:1347] Generated new cluster ID: 68a7dc5d6ee2417c9d58328afa544c53
I20260808 22:22:18.233778 19156 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:22:18.258644 19156 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:22:18.259684 19156 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:22:18.268306 19156 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be929da42de644648042b1c514d26d16: Generated new TSK 0
I20260808 22:22:18.269117 19156 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.9.176.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.176.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.176.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/krb5kdc/oryx_127.9.176.193.keytab.
Entry for principal oryx/127.9.176.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/krb5kdc/oryx_127.9.176.193.keytab.
I20260808 22:22:18.311781 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.193:0
--local_ip_for_outbound_sockets=127.9.176.193
--webserver_interface=127.9.176.193
--webserver_port=0
--tserver_master_addrs=127.9.176.254:46579
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/krb5kdc/oryx_127.9.176.193.keytab
--principal=oryx/127.9.176.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:43621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:22:18.442026 19173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:22:18.442276 19173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:22:18.442315 19173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:22:18.446193 19173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:22:18.446244 19173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:22:18.446298 19173 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.193
I20260808 22:22:18.451227 19173 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:43621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/krb5kdc/oryx_127.9.176.193.keytab
--principal=oryx/127.9.176.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.176.193
--webserver_port=0
--tserver_master_addrs=127.9.176.254:46579
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19173
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.193
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:22:18.452099 19173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:22:18.453019 19173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:22:18.461825 19182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:18.461880 19188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:18.461841 19180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:22:18.462003 19173 server_base.cc:1034] running on GCE node
I20260808 22:22:18.462456 19173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:22:18.463037 19173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:22:18.464198 19173 hybrid_clock.cc:648] HybridClock initialized: now 1786227738464183 us; error 25 us; skew 500 ppm
Aug 08 22:22:18 dist-test-slave-ncp9 krb5kdc[19010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227738, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.176.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:22:18.466717 19173 init.cc:377] Logged in from keytab as oryx/127.9.176.193@KRBTEST.COM (short username oryx)
I20260808 22:22:18.467517 19173 webserver.cc:460] Webserver started at http://127.9.176.193:41841/ using document root <none> and password file <none>
I20260808 22:22:18.468012 19173 fs_manager.cc:359] Metadata directory not provided
I20260808 22:22:18.468077 19173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:22:18.468282 19173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:22:18.470293 19173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-0/data/instance:
uuid: "39101d245faa4b6eb281ffc89ef69bf6"
format_stamp: "Formatted at 2026-08-08 22:22:18 on dist-test-slave-ncp9"
server_key: "c9342399e589036eda2af070e7c2a46d"
server_key_iv: "a5ea9178152bf8f6cb6bcb27f9edd517"
server_key_version: "encryptionkey@0"
I20260808 22:22:18.470819 19173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-0/wal/instance:
uuid: "39101d245faa4b6eb281ffc89ef69bf6"
format_stamp: "Formatted at 2026-08-08 22:22:18 on dist-test-slave-ncp9"
server_key: "c9342399e589036eda2af070e7c2a46d"
server_key_iv: "a5ea9178152bf8f6cb6bcb27f9edd517"
server_key_version: "encryptionkey@0"
I20260808 22:22:18.474304 19173 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 22:22:18.476444 19200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:18.477577 19173 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 22:22:18.477680 19173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-0/wal
uuid: "39101d245faa4b6eb281ffc89ef69bf6"
format_stamp: "Formatted at 2026-08-08 22:22:18 on dist-test-slave-ncp9"
server_key: "c9342399e589036eda2af070e7c2a46d"
server_key_iv: "a5ea9178152bf8f6cb6bcb27f9edd517"
server_key_version: "encryptionkey@0"
I20260808 22:22:18.477770 19173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:22:18.489740 19173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:22:18.492517 19173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:22:18.492682 19173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:22:18.493244 19173 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:22:18.493992 19173 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:22:18.494040 19173 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:18.494091 19173 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:22:18.494115 19173 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:18.502496 19173 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.193:34919
I20260808 22:22:18.502522 19387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.193:34919 every 8 connection(s)
I20260808 22:22:18.503336 19173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
I20260808 22:22:18.508577 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 19173
I20260808 22:22:18.508702 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-0/wal/instance
I20260808 22:22:18.508991 9923 external_mini_cluster.cc:1428] Setting key e31e09b3cfa32944f000da5acde88e47
Aug 08 22:22:18 dist-test-slave-ncp9 krb5kdc[19010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227738, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.176.193@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM
I20260808 22:22:18.516069 19143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:18.504984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.193:57051 (local address 127.9.176.254:46579)
0808 22:22:18.505133 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:18.505138 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:18.506165 (+ 1027us) server_negotiation.cc:407] Connection header received
0808 22:22:18.506947 (+ 782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:18.506950 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:18.506994 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:18.507065 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:18.508173 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:18.508699 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:18.509901 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:18.510137 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:18.510233 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:18.512646 (+ 2413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:18.512650 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:18.512663 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:22:18.512691 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:18.514132 (+ 1441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:18.514593 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:18.514597 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:18.514599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:18.514698 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:18.514997 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:18.515000 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:18.515002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:18.515233 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:18.515333 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:18.515780 (+ 447us) server_negotiation.cc:299] Negotiation successful
0808 22:22:18.515905 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 22:22:18.516745 19391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:18.505390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:46579 (local address 127.9.176.193:57051)
0808 22:22:18.506020 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:18.506062 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:18.506787 (+ 725us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:18.507211 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:18.507218 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:18.507597 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:18.508054 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:18.508067 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:18.508834 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:18.508837 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:18.509779 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:18.509790 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:18.510282 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:18.510286 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:18.510451 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:18.510955 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:18.510971 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:18.512472 (+ 1501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:22:18.514236 (+ 1764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:18.514240 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:18.514251 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:18.514498 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:18.514779 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:18.514781 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:18.514782 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:18.514907 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:18.515328 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:22:18.515333 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:22:18.515551 (+ 218us) client_negotiation.cc:769] Sending connection context
0808 22:22:18.515702 (+ 151us) client_negotiation.cc:240] Negotiation successful
0808 22:22:18.515934 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":109,"threads_started":1}
I20260808 22:22:18.517896 19389 heartbeater.cc:347] Connected to a master server at 127.9.176.254:46579
I20260808 22:22:18.518116 19389 heartbeater.cc:464] Registering TS with master...
I20260808 22:22:18.518544 19389 heartbeater.cc:511] Master 127.9.176.254:46579 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.9.176.194@KRBTEST.COM; defaulting to no policy
I20260808 22:22:18.519939 19078 ts_manager.cc:194] Registered new tserver with Master: 39101d245faa4b6eb281ffc89ef69bf6 (127.9.176.193:34919)
I20260808 22:22:18.521154 19078 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.176.193@KRBTEST.COM'} at 127.9.176.193:57051
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.176.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.176.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/krb5kdc/oryx_127.9.176.194.keytab.
Entry for principal oryx/127.9.176.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/krb5kdc/oryx_127.9.176.194.keytab.
I20260808 22:22:18.535660 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.194:0
--local_ip_for_outbound_sockets=127.9.176.194
--webserver_interface=127.9.176.194
--webserver_port=0
--tserver_master_addrs=127.9.176.254:46579
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/krb5kdc/oryx_127.9.176.194.keytab
--principal=oryx/127.9.176.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:43621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:22:18.644542 19395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:22:18.644786 19395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:22:18.644825 19395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:22:18.648658 19395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:22:18.648712 19395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:22:18.648770 19395 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.194
I20260808 22:22:18.653817 19395 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:43621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/krb5kdc/oryx_127.9.176.194.keytab
--principal=oryx/127.9.176.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.176.194
--webserver_port=0
--tserver_master_addrs=127.9.176.254:46579
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19395
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.194
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:22:18.654765 19395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:22:18.655723 19395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:22:18.665042 19409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:18.665021 19411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:18.665004 19407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:22:18.665239 19395 server_base.cc:1034] running on GCE node
I20260808 22:22:18.665634 19395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:22:18.666230 19395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:22:18.667388 19395 hybrid_clock.cc:648] HybridClock initialized: now 1786227738667369 us; error 32 us; skew 500 ppm
Aug 08 22:22:18 dist-test-slave-ncp9 krb5kdc[19010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227738, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.176.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:22:18.669930 19395 init.cc:377] Logged in from keytab as oryx/127.9.176.194@KRBTEST.COM (short username oryx)
I20260808 22:22:18.670742 19395 webserver.cc:460] Webserver started at http://127.9.176.194:45029/ using document root <none> and password file <none>
I20260808 22:22:18.671280 19395 fs_manager.cc:359] Metadata directory not provided
I20260808 22:22:18.671346 19395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:22:18.671543 19395 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:22:18.673502 19395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-1/data/instance:
uuid: "41b49dbbd6a04fce85c78e62b80ba81e"
format_stamp: "Formatted at 2026-08-08 22:22:18 on dist-test-slave-ncp9"
server_key: "0dba81c13d72389afd57459a60df36a5"
server_key_iv: "7af60a186758cc14fe6740c8446dc662"
server_key_version: "encryptionkey@0"
I20260808 22:22:18.674029 19395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-1/wal/instance:
uuid: "41b49dbbd6a04fce85c78e62b80ba81e"
format_stamp: "Formatted at 2026-08-08 22:22:18 on dist-test-slave-ncp9"
server_key: "0dba81c13d72389afd57459a60df36a5"
server_key_iv: "7af60a186758cc14fe6740c8446dc662"
server_key_version: "encryptionkey@0"
I20260808 22:22:18.677471 19395 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 22:22:18.679702 19419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:18.680842 19395 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 22:22:18.681002 19395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-1/wal
uuid: "41b49dbbd6a04fce85c78e62b80ba81e"
format_stamp: "Formatted at 2026-08-08 22:22:18 on dist-test-slave-ncp9"
server_key: "0dba81c13d72389afd57459a60df36a5"
server_key_iv: "7af60a186758cc14fe6740c8446dc662"
server_key_version: "encryptionkey@0"
I20260808 22:22:18.681095 19395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:22:18.704070 19395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:22:18.706702 19395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:22:18.706871 19395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:22:18.707423 19395 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:22:18.708217 19395 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:22:18.708263 19395 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:18.708305 19395 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:22:18.708333 19395 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:18.716631 19395 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.194:46339
I20260808 22:22:18.716658 19605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.194:46339 every 8 connection(s)
I20260808 22:22:18.717536 19395 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
I20260808 22:22:18.721675 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 19395
I20260808 22:22:18.721769 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-1/wal/instance
I20260808 22:22:18.722015 9923 external_mini_cluster.cc:1428] Setting key 2790abeb175812b0d77d6fb04af51c8f
Aug 08 22:22:18 dist-test-slave-ncp9 krb5kdc[19010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227738, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.176.194@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM
I20260808 22:22:18.730829 19143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:18.719273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.194:53891 (local address 127.9.176.254:46579)
0808 22:22:18.719435 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:18.719440 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:18.720389 (+ 949us) server_negotiation.cc:407] Connection header received
0808 22:22:18.721308 (+ 919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:18.721312 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:18.721369 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:18.721442 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:18.722747 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:18.723342 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:18.724548 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:18.724878 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:18.725001 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:18.727594 (+ 2593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:18.727597 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:18.727600 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:22:18.727626 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:18.728934 (+ 1308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:18.729408 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:18.729412 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:18.729414 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:18.729490 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:18.729800 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:18.729803 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:18.729805 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:18.729981 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:18.730074 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:18.730545 (+ 471us) server_negotiation.cc:299] Negotiation successful
0808 22:22:18.730669 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 22:22:18.731616 19608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:18.719633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:46579 (local address 127.9.176.194:53891)
0808 22:22:18.720251 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:18.720291 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:18.721112 (+ 821us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:18.721641 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:18.721647 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:18.722030 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:18.722618 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:18.722636 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:18.723462 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:18.723464 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:18.724432 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:18.724443 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:18.725032 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:18.725037 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:18.725258 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:18.725863 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:18.725884 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:18.727434 (+ 1550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:22:18.729045 (+ 1611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:18.729051 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:18.729066 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:18.729314 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:18.729579 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:18.729582 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:18.729584 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:18.729713 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:18.730080 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:22:18.730087 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:22:18.730322 (+ 235us) client_negotiation.cc:769] Sending connection context
0808 22:22:18.730485 (+ 163us) client_negotiation.cc:240] Negotiation successful
0808 22:22:18.730809 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":99,"threads_started":1}
WARNING: no policy specified for oryx/127.9.176.195@KRBTEST.COM; defaulting to no policy
I20260808 22:22:18.732717 19606 heartbeater.cc:347] Connected to a master server at 127.9.176.254:46579
I20260808 22:22:18.732963 19606 heartbeater.cc:464] Registering TS with master...
I20260808 22:22:18.733464 19606 heartbeater.cc:511] Master 127.9.176.254:46579 requested a full tablet report, sending...
I20260808 22:22:18.734458 19078 ts_manager.cc:194] Registered new tserver with Master: 41b49dbbd6a04fce85c78e62b80ba81e (127.9.176.194:46339)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.176.195@KRBTEST.COM" created.
I20260808 22:22:18.735146 19078 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.176.194@KRBTEST.COM'} at 127.9.176.194:53891
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.176.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/krb5kdc/oryx_127.9.176.195.keytab.
Entry for principal oryx/127.9.176.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/krb5kdc/oryx_127.9.176.195.keytab.
I20260808 22:22:18.749007 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.195:0
--local_ip_for_outbound_sockets=127.9.176.195
--webserver_interface=127.9.176.195
--webserver_port=0
--tserver_master_addrs=127.9.176.254:46579
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/krb5kdc/oryx_127.9.176.195.keytab
--principal=oryx/127.9.176.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:43621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:22:18.856755 19611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:22:18.857115 19611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:22:18.857180 19611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:22:18.861239 19611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:22:18.861292 19611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:22:18.861348 19611 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.195
I20260808 22:22:18.866429 19611 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:43621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/krb5kdc/oryx_127.9.176.195.keytab
--principal=oryx/127.9.176.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.176.195
--webserver_port=0
--tserver_master_addrs=127.9.176.254:46579
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19611
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.195
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:22:18.867362 19611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:22:18.868335 19611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:22:18.877467 19619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:18.877483 19620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:18.877471 19623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:22:18.877682 19611 server_base.cc:1034] running on GCE node
I20260808 22:22:18.878147 19611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:22:18.878703 19611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:22:18.879853 19611 hybrid_clock.cc:648] HybridClock initialized: now 1786227738879833 us; error 33 us; skew 500 ppm
Aug 08 22:22:18 dist-test-slave-ncp9 krb5kdc[19010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227738, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.176.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:22:18.882496 19611 init.cc:377] Logged in from keytab as oryx/127.9.176.195@KRBTEST.COM (short username oryx)
I20260808 22:22:18.883293 19611 webserver.cc:460] Webserver started at http://127.9.176.195:36331/ using document root <none> and password file <none>
I20260808 22:22:18.883774 19611 fs_manager.cc:359] Metadata directory not provided
I20260808 22:22:18.883831 19611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:22:18.884053 19611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:22:18.886004 19611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-2/data/instance:
uuid: "cf24739aa12a410fa19526e69e0c2d06"
format_stamp: "Formatted at 2026-08-08 22:22:18 on dist-test-slave-ncp9"
server_key: "b4f019a48a02e642252613ece5673125"
server_key_iv: "261c00c609d412364e5e228aec58cfeb"
server_key_version: "encryptionkey@0"
I20260808 22:22:18.886552 19611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-2/wal/instance:
uuid: "cf24739aa12a410fa19526e69e0c2d06"
format_stamp: "Formatted at 2026-08-08 22:22:18 on dist-test-slave-ncp9"
server_key: "b4f019a48a02e642252613ece5673125"
server_key_iv: "261c00c609d412364e5e228aec58cfeb"
server_key_version: "encryptionkey@0"
I20260808 22:22:18.889974 19611 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 22:22:18.892231 19632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:18.893399 19611 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 22:22:18.893501 19611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-2/wal
uuid: "cf24739aa12a410fa19526e69e0c2d06"
format_stamp: "Formatted at 2026-08-08 22:22:18 on dist-test-slave-ncp9"
server_key: "b4f019a48a02e642252613ece5673125"
server_key_iv: "261c00c609d412364e5e228aec58cfeb"
server_key_version: "encryptionkey@0"
I20260808 22:22:18.893587 19611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:22:18.909727 19611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:22:18.912276 19611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:22:18.912442 19611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:22:18.913036 19611 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:22:18.913821 19611 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:22:18.913870 19611 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:18.913923 19611 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:22:18.913947 19611 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:18.922271 19611 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.195:37359
I20260808 22:22:18.922299 19807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.195:37359 every 8 connection(s)
I20260808 22:22:18.923074 19611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
I20260808 22:22:18.924670 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 19611
I20260808 22:22:18.924783 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-2/wal/instance
I20260808 22:22:18.925113 9923 external_mini_cluster.cc:1428] Setting key 9eda338ea028cc680f0c39c6cf4d1b0f
Aug 08 22:22:18 dist-test-slave-ncp9 krb5kdc[19010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227738, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.176.195@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM
I20260808 22:22:18.935066 19143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:18.924790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.195:41399 (local address 127.9.176.254:46579)
0808 22:22:18.924960 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:18.924966 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:18.926082 (+ 1116us) server_negotiation.cc:407] Connection header received
0808 22:22:18.927035 (+ 953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:18.927038 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:18.927091 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:18.927173 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:18.928252 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:18.928769 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:18.929843 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:18.930050 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:18.930120 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:18.932339 (+ 2219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:18.932342 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:18.932343 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:22:18.932367 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:18.933514 (+ 1147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:18.933932 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:18.933935 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:18.933936 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:18.934002 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:18.934249 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:18.934251 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:18.934252 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:18.934411 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:18.934478 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:18.934858 (+ 380us) server_negotiation.cc:299] Negotiation successful
0808 22:22:18.934951 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 22:22:18.935801 19813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:18.925221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:46579 (local address 127.9.176.195:41399)
0808 22:22:18.925937 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:18.925992 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:18.926816 (+ 824us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:18.927326 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:18.927332 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:18.927675 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:18.928140 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:18.928153 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:18.928858 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:18.928861 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:18.929749 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:18.929758 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:18.930152 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:18.930155 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:18.930284 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:18.930818 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:18.930834 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:18.932200 (+ 1366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:22:18.933609 (+ 1409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:18.933614 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:18.933624 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:18.933856 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:18.934068 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:18.934070 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:18.934071 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:18.934187 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:18.934477 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:22:18.934482 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:22:18.934684 (+ 202us) client_negotiation.cc:769] Sending connection context
0808 22:22:18.934826 (+ 142us) client_negotiation.cc:240] Negotiation successful
0808 22:22:18.935063 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":130,"threads_started":1}
I20260808 22:22:18.936796 19809 heartbeater.cc:347] Connected to a master server at 127.9.176.254:46579
I20260808 22:22:18.937027 19809 heartbeater.cc:464] Registering TS with master...
I20260808 22:22:18.937482 19809 heartbeater.cc:511] Master 127.9.176.254:46579 requested a full tablet report, sending...
I20260808 22:22:18.938704 19077 ts_manager.cc:194] Registered new tserver with Master: cf24739aa12a410fa19526e69e0c2d06 (127.9.176.195:37359)
I20260808 22:22:18.939260 19077 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.176.195@KRBTEST.COM'} at 127.9.176.195:41399
I20260808 22:22:18.950174 9923 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.9.176.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.176.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.176.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/krb5kdc/oryx_127.9.176.253.keytab.
Entry for principal oryx/127.9.176.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/krb5kdc/oryx_127.9.176.253.keytab.
I20260808 22:22:18.975664 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.253:37039
--webserver_interface=127.9.176.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/krb5kdc/oryx_127.9.176.253.keytab
--principal=oryx/127.9.176.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:43621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.176.254:46579,127.9.176.253:37039
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:22:19.085879 19816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:22:19.086138 19816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:22:19.086193 19816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:22:19.090134 19816 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:22:19.090190 19816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:22:19.090219 19816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:22:19.090240 19816 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:22:19.090268 19816 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:22:19.095455 19816 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:43621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.9.176.254:46579,127.9.176.253:37039
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/krb5kdc/oryx_127.9.176.253.keytab
--principal=oryx/127.9.176.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.253:37039
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.9.176.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19816
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
W20260808 22:22:19.095686 19816 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.9.176.254:46579,127.9.176.253:37039'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260808 22:22:19.096573 19816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:22:19.097697 19816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:22:19.105773 19824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:22:19.105834 19816 server_base.cc:1034] running on GCE node
W20260808 22:22:19.105839 19827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:19.105773 19823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:22:19.106570 19816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:22:19.107579 19816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:22:19.108726 19816 hybrid_clock.cc:648] HybridClock initialized: now 1786227739108700 us; error 39 us; skew 500 ppm
Aug 08 22:22:19 dist-test-slave-ncp9 krb5kdc[19010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227739, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.176.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:22:19.111522 19816 init.cc:377] Logged in from keytab as oryx/127.9.176.253@KRBTEST.COM (short username oryx)
I20260808 22:22:19.112399 19816 webserver.cc:460] Webserver started at http://127.9.176.253:39657/ using document root <none> and password file <none>
I20260808 22:22:19.112957 19816 fs_manager.cc:359] Metadata directory not provided
I20260808 22:22:19.113023 19816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:22:19.113229 19816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:22:19.115223 19816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/master-1/data/instance:
uuid: "28f68f09769f4576ab963f9249cdf9e1"
format_stamp: "Formatted at 2026-08-08 22:22:19 on dist-test-slave-ncp9"
server_key: "fb476d8e856eac0609e1917a5c5ff197"
server_key_iv: "3ba0bb4d1567193505d02832d09f8368"
server_key_version: "encryptionkey@0"
I20260808 22:22:19.115768 19816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/master-1/wal/instance:
uuid: "28f68f09769f4576ab963f9249cdf9e1"
format_stamp: "Formatted at 2026-08-08 22:22:19 on dist-test-slave-ncp9"
server_key: "fb476d8e856eac0609e1917a5c5ff197"
server_key_iv: "3ba0bb4d1567193505d02832d09f8368"
server_key_version: "encryptionkey@0"
I20260808 22:22:19.119315 19816 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260808 22:22:19.121572 19837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:19.122720 19816 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 22:22:19.122833 19816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/master-1/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/master-1/wal
uuid: "28f68f09769f4576ab963f9249cdf9e1"
format_stamp: "Formatted at 2026-08-08 22:22:19 on dist-test-slave-ncp9"
server_key: "fb476d8e856eac0609e1917a5c5ff197"
server_key_iv: "3ba0bb4d1567193505d02832d09f8368"
server_key_version: "encryptionkey@0"
I20260808 22:22:19.122921 19816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:22:19.138208 19816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:22:19.141301 19816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:22:19.141470 19816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:22:19.148010 19816 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.253:37039
I20260808 22:22:19.148264 19927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.253:37039 every 8 connection(s)
I20260808 22:22:19.149050 19816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/master-1/data/info.pb
I20260808 22:22:19.151551 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 19816
I20260808 22:22:19.151791 19929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:22:19.156441 19929 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.176.254" port: 46579 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 22:22:19 dist-test-slave-ncp9 krb5kdc[19010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227737, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.176.253@KRBTEST.COM
Aug 08 22:22:19 dist-test-slave-ncp9 krb5kdc[19010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227739, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.176.253@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM
I20260808 22:22:19.166076 19936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:19.153119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36024 (local address 127.9.176.253:37039)
0808 22:22:19.153495 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:19.153510 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:19.153553 (+ 43us) server_negotiation.cc:407] Connection header received
0808 22:22:19.154337 (+ 784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:19.154355 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:19.154703 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:19.154976 (+ 273us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:19.155914 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:19.156991 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:19.158105 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:19.158773 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:19.159037 (+ 264us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:19.161485 (+ 2448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:19.161490 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:19.161499 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:22:19.161524 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:19.163185 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:19.163633 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:19.163638 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:19.163642 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:19.163756 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:19.164025 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:19.164028 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:19.164029 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:19.164479 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:19.164682 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:19.164963 (+ 281us) server_negotiation.cc:299] Negotiation successful
0808 22:22:19.165211 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":111,"threads_started":1}
I20260808 22:22:19.167826 19937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:19.158258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:46579 (local address 127.0.0.1:34106)
0808 22:22:19.158615 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:19.158659 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:19.158773 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:19.159179 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:19.159185 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:19.159597 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:19.159773 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:19.159782 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:19.160576 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:19.160579 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:19.161328 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:19.161338 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:19.161792 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:19.161795 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:19.161893 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:19.162307 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:19.162321 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:19.163820 (+ 1499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:22:19.165582 (+ 1762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:19.165589 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:19.165594 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:19.165854 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:19.166112 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:19.166115 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:19.166117 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:19.166208 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:19.167375 (+ 1167us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:22:19.167380 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:22:19.167473 (+ 93us) client_negotiation.cc:769] Sending connection context
0808 22:22:19.167578 (+ 105us) client_negotiation.cc:240] Negotiation successful
0808 22:22:19.167708 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":135,"threads_started":1}
I20260808 22:22:19.167932 19143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:19.158316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34106 (local address 127.9.176.254:46579)
0808 22:22:19.158427 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:19.158432 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:19.158738 (+ 306us) server_negotiation.cc:407] Connection header received
0808 22:22:19.158854 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:19.158857 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:19.158907 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:19.158975 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:19.159917 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:19.160445 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:19.161453 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:19.161688 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:19.161782 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:19.163954 (+ 2172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:19.163957 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:19.163959 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:22:19.163983 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:19.165480 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:19.165945 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:19.165948 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:19.165949 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:19.166024 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:19.166983 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:19.166987 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:19.166988 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:19.167274 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:19.167373 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:19.167659 (+ 286us) server_negotiation.cc:299] Negotiation successful
0808 22:22:19.167786 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260808 22:22:19.168918 19929 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.176.253" port: 37039 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 22:22:19 dist-test-slave-ncp9 krb5kdc[19010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227739, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.176.253@KRBTEST.COM for oryx/127.9.176.253@KRBTEST.COM
I20260808 22:22:19.176110 19937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:19.169326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.253:37039 (local address 127.0.0.1:36036)
0808 22:22:19.169431 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:19.169444 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:19.169505 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:19.169667 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:19.169669 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:19.169785 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:19.169867 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:19.169871 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:19.170486 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:19.170488 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:19.171035 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:19.171044 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:19.171416 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:19.171418 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:19.171484 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:19.171912 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:19.171923 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:19.173619 (+ 1696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:22:19.175070 (+ 1451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:19.175072 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:19.175074 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:19.175231 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:19.175419 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:19.175420 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:19.175422 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:19.175479 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:19.175729 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:22:19.175731 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:22:19.175800 (+ 69us) client_negotiation.cc:769] Sending connection context
0808 22:22:19.175870 (+ 70us) client_negotiation.cc:240] Negotiation successful
0808 22:22:19.175970 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260808 22:22:19.176136 19936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:19.169396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36036 (local address 127.9.176.253:37039)
0808 22:22:19.169489 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:19.169493 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:19.169506 (+ 13us) server_negotiation.cc:407] Connection header received
0808 22:22:19.169560 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:19.169562 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:19.169592 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:19.169650 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:19.169942 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:19.170407 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:19.171122 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:19.171341 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:19.171412 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:19.173745 (+ 2333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:19.173748 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:19.173750 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:22:19.173770 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:19.174993 (+ 1223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:19.175296 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:19.175298 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:19.175299 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:19.175361 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:19.175528 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:19.175530 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:19.175531 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:19.175670 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:19.175735 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:19.175900 (+ 165us) server_negotiation.cc:299] Negotiation successful
0808 22:22:19.176020 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":32}
I20260808 22:22:19.178799 19929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28f68f09769f4576ab963f9249cdf9e1: Bootstrap starting.
I20260808 22:22:19.181155 19929 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 28f68f09769f4576ab963f9249cdf9e1: Neither blocks nor log segments found. Creating new log.
I20260808 22:22:19.181897 19929 log.cc:824] T 00000000000000000000000000000000 P 28f68f09769f4576ab963f9249cdf9e1: Log is configured to *not* fsync() on all Append() calls
I20260808 22:22:19.183820 19929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28f68f09769f4576ab963f9249cdf9e1: No bootstrap required, opened a new log
I20260808 22:22:19.186899 19929 raft_consensus.cc:348] T 00000000000000000000000000000000 P 28f68f09769f4576ab963f9249cdf9e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be929da42de644648042b1c514d26d16" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 46579 } } peers { permanent_uuid: "28f68f09769f4576ab963f9249cdf9e1" member_type: VOTER last_known_addr { host: "127.9.176.253" port: 37039 } }
I20260808 22:22:19.187098 19929 raft_consensus.cc:374] T 00000000000000000000000000000000 P 28f68f09769f4576ab963f9249cdf9e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:22:19.187152 19929 raft_consensus.cc:729] T 00000000000000000000000000000000 P 28f68f09769f4576ab963f9249cdf9e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28f68f09769f4576ab963f9249cdf9e1, State: Initialized, Role: FOLLOWER
I20260808 22:22:19.187577 19929 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28f68f09769f4576ab963f9249cdf9e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be929da42de644648042b1c514d26d16" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 46579 } } peers { permanent_uuid: "28f68f09769f4576ab963f9249cdf9e1" member_type: VOTER last_known_addr { host: "127.9.176.253" port: 37039 } }
I20260808 22:22:19.188024 19940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28f68f09769f4576ab963f9249cdf9e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be929da42de644648042b1c514d26d16" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 46579 } } peers { permanent_uuid: "28f68f09769f4576ab963f9249cdf9e1" member_type: VOTER last_known_addr { host: "127.9.176.253" port: 37039 } } }
I20260808 22:22:19.188133 19940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28f68f09769f4576ab963f9249cdf9e1 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 22:22:19.188727 19929 sys_catalog.cc:564] T 00000000000000000000000000000000 P 28f68f09769f4576ab963f9249cdf9e1 [sys.catalog]: configured and running, proceeding with master startup.
W20260808 22:22:19.193100 19959 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 28f68f09769f4576ab963f9249cdf9e1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 22:22:19.193192 19959 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 22:22:19.195336 19816 master_runner.cc:428] Detected that this master 28f68f09769f4576ab963f9249cdf9e1 is joining an existing cluster
I20260808 22:22:19.195399 19816 master_runner.cc:432] Initiating AddMaster RPC to add 127.9.176.253:37039
I20260808 22:22:19.207448 19936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:19.198190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36044 (local address 127.9.176.253:37039)
0808 22:22:19.198296 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:19.198300 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:19.198531 (+ 231us) server_negotiation.cc:407] Connection header received
0808 22:22:19.198642 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:19.198645 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:19.198677 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:19.198741 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:19.199350 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:19.200113 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:19.202594 (+ 2481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:19.202914 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:19.202985 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:19.204247 (+ 1262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:19.204250 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:19.204253 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:22:19.204274 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:19.205765 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:19.206311 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:19.206315 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:19.206317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:19.206390 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:19.206637 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:19.206640 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:19.206641 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:19.206829 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:19.206890 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:19.207182 (+ 292us) server_negotiation.cc:299] Negotiation successful
0808 22:22:19.207291 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37,"spinlock_wait_cycles":1280}
I20260808 22:22:19.207578 19143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:19.197740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34110 (local address 127.9.176.254:46579)
0808 22:22:19.197879 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:19.197883 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:19.198459 (+ 576us) server_negotiation.cc:407] Connection header received
0808 22:22:19.198596 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:19.198599 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:19.198659 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:19.198730 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:19.199329 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:19.200115 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:19.201766 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:19.202071 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:19.202192 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:19.204043 (+ 1851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:19.204047 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:19.204050 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:22:19.204075 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:19.205768 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:19.206329 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:19.206332 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:19.206334 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:19.206414 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:19.206687 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:19.206690 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:19.206692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:19.206889 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:19.206956 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:19.207323 (+ 367us) server_negotiation.cc:299] Negotiation successful
0808 22:22:19.207435 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 22:22:19.207849 19971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:19.198092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.253:37039 (local address 127.0.0.1:36044)
0808 22:22:19.198445 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:19.198458 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:19.198560 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:19.198772 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:19.198775 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:19.199078 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:19.199259 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:19.199267 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:19.200232 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:19.200235 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:19.202473 (+ 2238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:19.202484 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:19.203126 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:19.203129 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:19.203234 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:19.203657 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:19.203671 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:19.204121 (+ 450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:22:19.205875 (+ 1754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:19.205877 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:19.205880 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:19.206219 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:19.206476 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:19.206478 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:19.206480 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:19.206553 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:19.206942 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:22:19.206945 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:22:19.207014 (+ 69us) client_negotiation.cc:769] Sending connection context
0808 22:22:19.207570 (+ 556us) client_negotiation.cc:240] Negotiation successful
0808 22:22:19.207690 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"mutex_wait_us":58,"thread_start_us":98,"threads_started":1}
I20260808 22:22:19.208375 19970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:19.197673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:46579 (local address 127.0.0.1:34110)
0808 22:22:19.198376 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:19.198388 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:19.198479 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:19.198842 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:19.198845 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:19.199080 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:19.199218 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:19.199223 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:19.200682 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:19.200685 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:19.201626 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:19.201638 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:19.202320 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:19.202324 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:19.202506 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:19.203384 (+ 878us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:19.203401 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:19.203904 (+ 503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:22:19.205885 (+ 1981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:19.205888 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:19.205890 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:19.206227 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:19.206511 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:19.206514 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:19.206516 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:19.206593 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:19.207028 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:22:19.207031 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:22:19.207133 (+ 102us) client_negotiation.cc:769] Sending connection context
0808 22:22:19.208072 (+ 939us) client_negotiation.cc:240] Negotiation successful
0808 22:22:19.208214 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"mutex_wait_us":54,"thread_start_us":85,"threads_started":1}
Aug 08 22:22:19 dist-test-slave-ncp9 krb5kdc[19010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227738, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.176.254@KRBTEST.COM for oryx/127.9.176.253@KRBTEST.COM
I20260808 22:22:19.221155 19940 raft_consensus.cc:482] T 00000000000000000000000000000000 P 28f68f09769f4576ab963f9249cdf9e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 22:22:19.221325 19940 raft_consensus.cc:504] T 00000000000000000000000000000000 P 28f68f09769f4576ab963f9249cdf9e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be929da42de644648042b1c514d26d16" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 46579 } } peers { permanent_uuid: "28f68f09769f4576ab963f9249cdf9e1" member_type: VOTER last_known_addr { host: "127.9.176.253" port: 37039 } }
I20260808 22:22:19.222287 19940 leader_election.cc:288] T 00000000000000000000000000000000 P 28f68f09769f4576ab963f9249cdf9e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers be929da42de644648042b1c514d26d16 (127.9.176.254:46579)
I20260808 22:22:19.222821 19096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "28f68f09769f4576ab963f9249cdf9e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be929da42de644648042b1c514d26d16" is_pre_election: true
I20260808 22:22:19.223206 19936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:19.213056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36052 (local address 127.9.176.253:37039)
0808 22:22:19.213149 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:19.213161 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:19.213505 (+ 344us) server_negotiation.cc:407] Connection header received
0808 22:22:19.213742 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:19.213745 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:19.213796 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:19.213872 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:19.214599 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:19.215358 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:19.216461 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:19.216795 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:19.216913 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:19.219466 (+ 2553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:19.219470 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:19.219473 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:22:19.219497 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:19.221079 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:19.221646 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:19.221650 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:19.221664 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:19.221756 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:19.222080 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:19.222083 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:19.222084 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:19.222371 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:19.222439 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:19.222923 (+ 484us) server_negotiation.cc:299] Negotiation successful
0808 22:22:19.223052 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260808 22:22:19.223557 19839 leader_election.cc:302] T 00000000000000000000000000000000 P 28f68f09769f4576ab963f9249cdf9e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 28f68f09769f4576ab963f9249cdf9e1; no voters: be929da42de644648042b1c514d26d16
I20260808 22:22:19.223726 19972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:19.212940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.253:37039 (local address 127.0.0.1:36052)
0808 22:22:19.213392 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:19.213442 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:19.213596 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:19.213907 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:19.213912 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:19.214343 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:19.214490 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:19.214499 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:19.215458 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:19.215462 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:19.216353 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:19.216365 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:19.216886 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:19.216909 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:19.217005 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:19.217492 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:19.217503 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:19.219148 (+ 1645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:22:19.221203 (+ 2055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:19.221214 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:19.221222 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:19.221522 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:19.221876 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:19.221879 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:19.221881 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:19.221973 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:19.222524 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:22:19.222531 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:22:19.222682 (+ 151us) client_negotiation.cc:769] Sending connection context
0808 22:22:19.223338 (+ 656us) client_negotiation.cc:240] Negotiation successful
0808 22:22:19.223558 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":95,"threads_started":1}
I20260808 22:22:19.224016 19940 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 28f68f09769f4576ab963f9249cdf9e1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 22:22:19.225370 19077 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.9.176.253:37039: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "28f68f09769f4576ab963f9249cdf9e1"
member_type: NON_VOTER
last_known_addr {
host: "127.9.176.253"
port: 37039
}
attrs {
promote: true
}
}
dest_uuid: "be929da42de644648042b1c514d26d16"
cas_config_opid_index: -1
I20260808 22:22:19.226114 19077 consensus_queue.cc:237] T 00000000000000000000000000000000 P be929da42de644648042b1c514d26d16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "be929da42de644648042b1c514d26d16" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 46579 } } peers { permanent_uuid: "28f68f09769f4576ab963f9249cdf9e1" member_type: NON_VOTER last_known_addr { host: "127.9.176.253" port: 37039 } attrs { promote: true } }
I20260808 22:22:19.228134 19974 raft_consensus.cc:2932] T 00000000000000000000000000000000 P be929da42de644648042b1c514d26d16 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 28f68f09769f4576ab963f9249cdf9e1 (127.9.176.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "be929da42de644648042b1c514d26d16" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 46579 } } peers { permanent_uuid: "28f68f09769f4576ab963f9249cdf9e1" member_type: NON_VOTER last_known_addr { host: "127.9.176.253" port: 37039 } attrs { promote: true } } }
I20260808 22:22:19.228724 19880 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 28f68f09769f4576ab963f9249cdf9e1 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:22:19.229786 19974 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.9.176.253:37039
I20260808 22:22:19.229838 19880 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 28f68f09769f4576ab963f9249cdf9e1 [term 1 FOLLOWER]: Refusing update from remote peer be929da42de644648042b1c514d26d16: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260808 22:22:19.230260 19816 master.cc:537] Master@127.9.176.253:37039 shutting down...
WARNING: no policy specified for oryx/127.9.176.252@KRBTEST.COM; defaulting to no policy
I20260808 22:22:19.234341 19974 sys_catalog.cc:455] T 00000000000000000000000000000000 P be929da42de644648042b1c514d26d16 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "be929da42de644648042b1c514d26d16" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "be929da42de644648042b1c514d26d16" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 46579 } } peers { permanent_uuid: "28f68f09769f4576ab963f9249cdf9e1" member_type: NON_VOTER last_known_addr { host: "127.9.176.253" port: 37039 } attrs { promote: true } } }
I20260808 22:22:19.234457 19974 sys_catalog.cc:458] T 00000000000000000000000000000000 P be929da42de644648042b1c514d26d16 [sys.catalog]: This master's current role is: LEADER
I20260808 22:22:19.234653 19974 consensus_queue.cc:1035] T 00000000000000000000000000000000 P be929da42de644648042b1c514d26d16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28f68f09769f4576ab963f9249cdf9e1" member_type: NON_VOTER last_known_addr { host: "127.9.176.253" port: 37039 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:22:19.235442 19816 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 28f68f09769f4576ab963f9249cdf9e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:22:19.235711 19816 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 28f68f09769f4576ab963f9249cdf9e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:22:19.235811 19816 tablet_replica.cc:330] T 00000000000000000000000000000000 P 28f68f09769f4576ab963f9249cdf9e1: stopping tablet replica
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.176.252@KRBTEST.COM" created.
W20260808 22:22:19.237205 19053 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260808 22:22:19.238833 19839 connection.cc:441] server connection from 127.0.0.1:36052 torn down before Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36052 (request call id 3) could send its response
W20260808 22:22:19.239420 19053 consensus_peers.cc:487] T 00000000000000000000000000000000 P be929da42de644648042b1c514d26d16 -> Peer 28f68f09769f4576ab963f9249cdf9e1 (127.9.176.253:37039): Couldn't send request to peer 28f68f09769f4576ab963f9249cdf9e1. Status: Network error: recv got EOF from 127.9.176.253:37039 (error 108). This is attempt 1: this message will repeat every 5th retry.
I20260808 22:22:19.241415 19816 master.cc:552] Master@127.9.176.253:37039 shutdown complete.
I20260808 22:22:19.241456 19816 master_runner.cc:305] Clearing existing system tablet
I20260808 22:22:19.241675 19816 fs_manager.cc:359] Metadata directory not provided
I20260808 22:22:19.241729 19816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:22:19.242777 19816 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 22:22:19.243510 19986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:19.243709 19816 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:22:19.243830 19816 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 22:22:19.243868 19816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/master-1/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/master-1/wal
uuid: "28f68f09769f4576ab963f9249cdf9e1"
format_stamp: "Formatted at 2026-08-08 22:22:19 on dist-test-slave-ncp9"
server_key: "fb476d8e856eac0609e1917a5c5ff197"
server_key_iv: "3ba0bb4d1567193505d02832d09f8368"
server_key_version: "encryptionkey@0"
I20260808 22:22:19.245533 19816 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 28f68f09769f4576ab963f9249cdf9e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:22:19.247308 19816 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 28f68f09769f4576ab963f9249cdf9e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260808 22:22:19.247392 19816 log.cc:1196] T 00000000000000000000000000000000 P 28f68f09769f4576ab963f9249cdf9e1: Deleting WAL directory at /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260808 22:22:19.247604 19816 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 28f68f09769f4576ab963f9249cdf9e1: Deleting consensus metadata
I20260808 22:22:19.247761 19816 master_runner.cc:315] Copying system tablet from 127.9.176.254:46579
I20260808 22:22:19.248957 19816 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 28f68f09769f4576ab963f9249cdf9e1: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.254:46579
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.176.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/krb5kdc/oryx_127.9.176.252.keytab.
Entry for principal oryx/127.9.176.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/krb5kdc/oryx_127.9.176.252.keytab.
I20260808 22:22:19.256044 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.252:34903
--webserver_interface=127.9.176.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/krb5kdc/oryx_127.9.176.252.keytab
--principal=oryx/127.9.176.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:43621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.176.254:46579,127.9.176.253:37039,127.9.176.252:34903
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 22:22:19.257365 19992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:19.249534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:46579 (local address 127.0.0.1:34122)
0808 22:22:19.249876 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:19.249888 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:19.249984 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:19.250217 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:19.250219 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:19.250394 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:19.250486 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:19.250492 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:19.251565 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:19.251567 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:19.252150 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:19.252167 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:19.252724 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:19.252727 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:19.252803 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:19.253365 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:19.253376 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:19.253702 (+ 326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:22:19.255559 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:19.255563 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:19.255565 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:19.255814 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:19.256098 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:19.256102 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:19.256104 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:19.256213 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:19.256705 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:22:19.256708 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:22:19.256805 (+ 97us) client_negotiation.cc:769] Sending connection context
0808 22:22:19.257010 (+ 205us) client_negotiation.cc:240] Negotiation successful
0808 22:22:19.257169 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":82,"threads_started":1}
I20260808 22:22:19.257395 19143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:19.249623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34122 (local address 127.9.176.254:46579)
0808 22:22:19.249760 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:19.249765 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:19.249961 (+ 196us) server_negotiation.cc:407] Connection header received
0808 22:22:19.250064 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:19.250067 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:19.250119 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:19.250211 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:19.250591 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:19.251409 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:19.252270 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:19.252620 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:19.252721 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:19.253839 (+ 1118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:19.253843 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:19.253846 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:22:19.253875 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:19.255429 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:19.255909 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:19.255913 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:19.255915 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:19.256004 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:19.256331 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:19.256334 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:19.256336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:19.256558 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:19.256630 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:19.257114 (+ 484us) server_negotiation.cc:299] Negotiation successful
0808 22:22:19.257238 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 22:22:19.258438 19112 tablet_copy_service.cc:140] P be929da42de644648042b1c514d26d16: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 28f68f09769f4576ab963f9249cdf9e1 ({username='oryx', principal='oryx/127.9.176.253@KRBTEST.COM'} at 127.0.0.1:34122)
I20260808 22:22:19.258600 19112 tablet_copy_service.cc:161] P be929da42de644648042b1c514d26d16: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 28f68f09769f4576ab963f9249cdf9e1 at {username='oryx', principal='oryx/127.9.176.253@KRBTEST.COM'} at 127.0.0.1:34122: session id = 28f68f09769f4576ab963f9249cdf9e1-00000000000000000000000000000000
I20260808 22:22:19.260583 19112 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P be929da42de644648042b1c514d26d16: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 22:22:19.262740 19816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260808 22:22:19.265359 19816 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 28f68f09769f4576ab963f9249cdf9e1: tablet copy: Starting download of 0 data blocks...
I20260808 22:22:19.265566 19816 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 28f68f09769f4576ab963f9249cdf9e1: tablet copy: Starting download of 1 WAL segments...
I20260808 22:22:19.267558 19816 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 28f68f09769f4576ab963f9249cdf9e1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 22:22:19.269395 19112 tablet_copy_service.cc:342] P be929da42de644648042b1c514d26d16: Request end of tablet copy session 28f68f09769f4576ab963f9249cdf9e1-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.9.176.253@KRBTEST.COM'} at 127.0.0.1:34122
I20260808 22:22:19.269495 19112 tablet_copy_service.cc:433] P be929da42de644648042b1c514d26d16: ending tablet copy session 28f68f09769f4576ab963f9249cdf9e1-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 28f68f09769f4576ab963f9249cdf9e1
I20260808 22:22:19.277370 19816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:22:19.278992 19994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:19.281885 19995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:22:19.282019 19816 server_base.cc:1034] running on GCE node
W20260808 22:22:19.285007 19997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:22:19.285243 19816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:22:19.285544 19816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:22:19.286731 19816 hybrid_clock.cc:648] HybridClock initialized: now 1786227739286719 us; error 32 us; skew 500 ppm
Aug 08 22:22:19 dist-test-slave-ncp9 krb5kdc[19010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227739, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.176.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:22:19.288275 19816 init.cc:377] Logged in from keytab as oryx/127.9.176.253@KRBTEST.COM (short username oryx)
I20260808 22:22:19.288749 19816 webserver.cc:460] Webserver started at http://127.9.176.253:38695/ using document root <none> and password file <none>
I20260808 22:22:19.288959 19816 fs_manager.cc:359] Metadata directory not provided
I20260808 22:22:19.289003 19816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:22:19.290318 19816 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260808 22:22:19.291304 20009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:19.291538 19816 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 22:22:19.291611 19816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/master-1/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/master-1/wal
uuid: "28f68f09769f4576ab963f9249cdf9e1"
format_stamp: "Formatted at 2026-08-08 22:22:19 on dist-test-slave-ncp9"
server_key: "fb476d8e856eac0609e1917a5c5ff197"
server_key_iv: "3ba0bb4d1567193505d02832d09f8368"
server_key_version: "encryptionkey@0"
I20260808 22:22:19.291692 19816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:22:19.319159 19816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:22:19.321372 19816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:22:19.330574 19816 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.253:37039
I20260808 22:22:19.330607 20093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.253:37039 every 8 connection(s)
I20260808 22:22:19.331174 19816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/master-1/data/info.pb
I20260808 22:22:19.333189 20095 sys_catalog.cc:263] Verifying existing consensus state
I20260808 22:22:19.334250 20095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28f68f09769f4576ab963f9249cdf9e1: Bootstrap starting.
I20260808 22:22:19.344596 20095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28f68f09769f4576ab963f9249cdf9e1: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 22:22:19.344872 20095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28f68f09769f4576ab963f9249cdf9e1: Bootstrap complete.
I20260808 22:22:19.345397 20095 raft_consensus.cc:348] T 00000000000000000000000000000000 P 28f68f09769f4576ab963f9249cdf9e1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "be929da42de644648042b1c514d26d16" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 46579 } } peers { permanent_uuid: "28f68f09769f4576ab963f9249cdf9e1" member_type: NON_VOTER last_known_addr { host: "127.9.176.253" port: 37039 } attrs { promote: true } }
I20260808 22:22:19.345525 20095 raft_consensus.cc:729] T 00000000000000000000000000000000 P 28f68f09769f4576ab963f9249cdf9e1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 28f68f09769f4576ab963f9249cdf9e1, State: Initialized, Role: LEARNER
I20260808 22:22:19.345615 20095 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28f68f09769f4576ab963f9249cdf9e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "be929da42de644648042b1c514d26d16" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 46579 } } peers { permanent_uuid: "28f68f09769f4576ab963f9249cdf9e1" member_type: NON_VOTER last_known_addr { host: "127.9.176.253" port: 37039 } attrs { promote: true } }
I20260808 22:22:19.345920 20098 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28f68f09769f4576ab963f9249cdf9e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "be929da42de644648042b1c514d26d16" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 46579 } } peers { permanent_uuid: "28f68f09769f4576ab963f9249cdf9e1" member_type: NON_VOTER last_known_addr { host: "127.9.176.253" port: 37039 } attrs { promote: true } } }
I20260808 22:22:19.346032 20095 sys_catalog.cc:564] T 00000000000000000000000000000000 P 28f68f09769f4576ab963f9249cdf9e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:22:19.346127 20098 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28f68f09769f4576ab963f9249cdf9e1 [sys.catalog]: This master's current role is: LEARNER
I20260808 22:22:19.348740 20120 catalog_manager.cc:1259] Loaded cluster ID: 68a7dc5d6ee2417c9d58328afa544c53
I20260808 22:22:19.348852 20120 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 28f68f09769f4576ab963f9249cdf9e1: loading cluster ID for follower catalog manager: success
I20260808 22:22:19.351120 20120 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 28f68f09769f4576ab963f9249cdf9e1: acquiring CA information for follower catalog manager: success
I20260808 22:22:19.352517 20120 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 28f68f09769f4576ab963f9249cdf9e1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260808 22:22:19.391511 19993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:22:19.391782 19993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:22:19.391835 19993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:22:19.395841 19993 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:22:19.395900 19993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:22:19.395918 19993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:22:19.395937 19993 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:22:19.395965 19993 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:22:19.401233 19993 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:43621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.9.176.254:46579,127.9.176.253:37039,127.9.176.252:34903
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/krb5kdc/oryx_127.9.176.252.keytab
--principal=oryx/127.9.176.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.252:34903
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.9.176.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19993
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:22:19.402266 19993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:22:19.403272 19993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:22:19.411547 20125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:19.411590 20127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:19.411540 20129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:22:19.411649 19993 server_base.cc:1034] running on GCE node
I20260808 22:22:19.412326 19993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:22:19.413394 19993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:22:19.414538 19993 hybrid_clock.cc:648] HybridClock initialized: now 1786227739414515 us; error 40 us; skew 500 ppm
Aug 08 22:22:19 dist-test-slave-ncp9 krb5kdc[19010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227739, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.176.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:22:19.417187 19993 init.cc:377] Logged in from keytab as oryx/127.9.176.252@KRBTEST.COM (short username oryx)
I20260808 22:22:19.418429 19993 webserver.cc:460] Webserver started at http://127.9.176.252:39277/ using document root <none> and password file <none>
I20260808 22:22:19.418948 19993 fs_manager.cc:359] Metadata directory not provided
I20260808 22:22:19.419008 19993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:22:19.419224 19993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:22:19.421252 19993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/master-2/data/instance:
uuid: "bacae03153b74596ba962b8db7fedc8f"
format_stamp: "Formatted at 2026-08-08 22:22:19 on dist-test-slave-ncp9"
server_key: "3602e939c96672bd7fcd1765d8676dd2"
server_key_iv: "93fa90579f4157de3088f4e139613fba"
server_key_version: "encryptionkey@0"
I20260808 22:22:19.421782 19993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/master-2/wal/instance:
uuid: "bacae03153b74596ba962b8db7fedc8f"
format_stamp: "Formatted at 2026-08-08 22:22:19 on dist-test-slave-ncp9"
server_key: "3602e939c96672bd7fcd1765d8676dd2"
server_key_iv: "93fa90579f4157de3088f4e139613fba"
server_key_version: "encryptionkey@0"
I20260808 22:22:19.425652 19993 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 22:22:19.427996 20140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:19.429207 19993 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 22:22:19.429324 19993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/master-2/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/master-2/wal
uuid: "bacae03153b74596ba962b8db7fedc8f"
format_stamp: "Formatted at 2026-08-08 22:22:19 on dist-test-slave-ncp9"
server_key: "3602e939c96672bd7fcd1765d8676dd2"
server_key_iv: "93fa90579f4157de3088f4e139613fba"
server_key_version: "encryptionkey@0"
I20260808 22:22:19.429411 19993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:22:19.441066 19993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:22:19.443791 19993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:22:19.443944 19993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:22:19.455499 19993 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.252:34903
I20260808 22:22:19.455752 20225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.252:34903 every 8 connection(s)
I20260808 22:22:19.456444 19993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/master-2/data/info.pb
I20260808 22:22:19.459225 20228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:22:19.463279 20228 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.176.254" port: 46579 } has no permanent_uuid. Determining permanent_uuid...
I20260808 22:22:19.463973 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 19993
Aug 08 22:22:19 dist-test-slave-ncp9 krb5kdc[19010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227739, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.176.252@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM
Aug 08 22:22:19 dist-test-slave-ncp9 krb5kdc[19010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227737, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.176.252@KRBTEST.COM
I20260808 22:22:19.476511 19143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:19.465269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34128 (local address 127.9.176.254:46579)
0808 22:22:19.465400 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:19.465405 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:19.466216 (+ 811us) server_negotiation.cc:407] Connection header received
0808 22:22:19.467144 (+ 928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:19.467147 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:19.467199 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:19.467277 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:19.468229 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:19.468817 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:19.469971 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:19.470285 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:19.470363 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:19.473075 (+ 2712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:19.473079 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:19.473082 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:22:19.473110 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:19.474582 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:19.475081 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:19.475085 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:19.475087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:19.475172 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:19.475480 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:19.475483 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:19.475484 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:19.475669 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:19.475754 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:19.476234 (+ 480us) server_negotiation.cc:299] Negotiation successful
0808 22:22:19.476352 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260808 22:22:19.477341 20231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:19.465565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:46579 (local address 127.0.0.1:34128)
0808 22:22:19.466081 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:19.466122 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:19.466943 (+ 821us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:19.467349 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:19.467354 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:19.467799 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:19.468078 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:19.468090 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:19.468958 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:19.468962 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:19.469863 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:19.469872 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:19.470406 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:19.470410 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:19.470738 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:19.471341 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:19.471357 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:19.472936 (+ 1579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:22:19.474680 (+ 1744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:19.474688 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:19.474699 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:19.474988 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:19.475251 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:19.475254 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:19.475257 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:19.475391 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:19.475767 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:22:19.475773 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:22:19.476003 (+ 230us) client_negotiation.cc:769] Sending connection context
0808 22:22:19.476170 (+ 167us) client_negotiation.cc:240] Negotiation successful
0808 22:22:19.476459 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"mutex_wait_us":46,"thread_start_us":195,"threads_started":1}
I20260808 22:22:19.477397 20232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:19.466404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37038 (local address 127.9.176.252:34903)
0808 22:22:19.466738 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:19.466749 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:19.466782 (+ 33us) server_negotiation.cc:407] Connection header received
0808 22:22:19.466941 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:19.466956 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:19.467131 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:19.467282 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:19.468010 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:19.469355 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:19.470252 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:19.470820 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:19.470916 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:19.473482 (+ 2566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:19.473488 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:19.473497 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:22:19.473517 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:19.475119 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:19.475520 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:19.475525 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:19.475526 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:19.475600 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:19.475856 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:19.475859 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:19.475861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:19.476079 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:19.476315 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:19.476473 (+ 158us) server_negotiation.cc:299] Negotiation successful
0808 22:22:19.476600 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":56,"server-negotiator.queue_time_us":238,"thread_start_us":159,"threads_started":1}
I20260808 22:22:19.478735 20228 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.176.253" port: 37039 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 22:22:19 dist-test-slave-ncp9 krb5kdc[19010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227739, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.176.252@KRBTEST.COM for oryx/127.9.176.253@KRBTEST.COM
I20260808 22:22:19.486413 20231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:19.479209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.253:37039 (local address 127.0.0.1:36058)
0808 22:22:19.479307 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:19.479320 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:19.479385 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:19.479798 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:19.479800 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:19.479941 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:19.480118 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:19.480125 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:19.481011 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:19.481015 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:19.481584 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:19.481593 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:19.482029 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:19.482033 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:19.482127 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:19.482519 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:19.482532 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:19.483582 (+ 1050us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:22:19.485106 (+ 1524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:19.485109 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:19.485111 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:19.485306 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:19.485561 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:19.485564 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:19.485566 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:19.485639 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:19.485981 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:22:19.485984 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:22:19.486070 (+ 86us) client_negotiation.cc:769] Sending connection context
0808 22:22:19.486153 (+ 83us) client_negotiation.cc:240] Negotiation successful
0808 22:22:19.486272 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260808 22:22:19.486467 20235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:19.479297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36058 (local address 127.9.176.253:37039)
0808 22:22:19.479600 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:19.479604 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:19.479615 (+ 11us) server_negotiation.cc:407] Connection header received
0808 22:22:19.479650 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:19.479652 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:19.479691 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:19.479766 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:19.480238 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:19.480922 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:19.481696 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:19.481936 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:19.482021 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:19.483707 (+ 1686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:19.483711 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:19.483713 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:22:19.483737 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:19.484983 (+ 1246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:19.485389 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:19.485392 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:19.485394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:19.485472 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:19.485719 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:19.485722 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:19.485723 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:19.485893 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:19.485976 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:19.486216 (+ 240us) server_negotiation.cc:299] Negotiation successful
0808 22:22:19.486327 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":92,"threads_started":1}
I20260808 22:22:19.487164 20228 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.176.252" port: 34903 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 22:22:19 dist-test-slave-ncp9 krb5kdc[19010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227739, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.176.252@KRBTEST.COM for oryx/127.9.176.252@KRBTEST.COM
I20260808 22:22:19.493640 20231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:19.487550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.252:34903 (local address 127.0.0.1:37050)
0808 22:22:19.487641 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:19.487653 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:19.487722 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:19.487913 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:19.487916 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:19.488043 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:19.488104 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:19.488108 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:19.488753 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:19.488756 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:19.489296 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:19.489305 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:19.489699 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:19.489702 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:19.489770 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:19.490119 (+ 349us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:19.490130 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:19.491210 (+ 1080us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:22:19.492506 (+ 1296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:19.492509 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:19.492511 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:19.492689 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:19.492884 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:19.492886 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:19.492902 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:19.492983 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:19.493279 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:22:19.493281 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:22:19.493349 (+ 68us) client_negotiation.cc:769] Sending connection context
0808 22:22:19.493417 (+ 68us) client_negotiation.cc:240] Negotiation successful
0808 22:22:19.493509 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260808 22:22:19.493683 20232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:19.487604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37050 (local address 127.9.176.252:34903)
0808 22:22:19.487705 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:19.487708 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:19.487720 (+ 12us) server_negotiation.cc:407] Connection header received
0808 22:22:19.487804 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:19.487807 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:19.487837 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:19.487893 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:19.488196 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:19.488662 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:19.489394 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:19.489610 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:19.489699 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:19.491324 (+ 1625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:19.491326 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:19.491328 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:22:19.491348 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:19.492401 (+ 1053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:19.492758 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:19.492760 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:19.492761 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:19.492823 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:19.493057 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:19.493059 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:19.493060 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:19.493210 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:19.493276 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:19.493456 (+ 180us) server_negotiation.cc:299] Negotiation successful
0808 22:22:19.493561 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":35}
I20260808 22:22:19.496769 20228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bacae03153b74596ba962b8db7fedc8f: Bootstrap starting.
I20260808 22:22:19.499480 20228 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bacae03153b74596ba962b8db7fedc8f: Neither blocks nor log segments found. Creating new log.
I20260808 22:22:19.500186 20228 log.cc:824] T 00000000000000000000000000000000 P bacae03153b74596ba962b8db7fedc8f: Log is configured to *not* fsync() on all Append() calls
I20260808 22:22:19.502020 20228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bacae03153b74596ba962b8db7fedc8f: No bootstrap required, opened a new log
I20260808 22:22:19.505497 20228 raft_consensus.cc:348] T 00000000000000000000000000000000 P bacae03153b74596ba962b8db7fedc8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be929da42de644648042b1c514d26d16" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 46579 } } peers { permanent_uuid: "28f68f09769f4576ab963f9249cdf9e1" member_type: VOTER last_known_addr { host: "127.9.176.253" port: 37039 } } peers { permanent_uuid: "bacae03153b74596ba962b8db7fedc8f" member_type: VOTER last_known_addr { host: "127.9.176.252" port: 34903 } }
I20260808 22:22:19.505698 20228 raft_consensus.cc:374] T 00000000000000000000000000000000 P bacae03153b74596ba962b8db7fedc8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:22:19.505759 20228 raft_consensus.cc:729] T 00000000000000000000000000000000 P bacae03153b74596ba962b8db7fedc8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bacae03153b74596ba962b8db7fedc8f, State: Initialized, Role: FOLLOWER
I20260808 22:22:19.506155 20228 consensus_queue.cc:260] T 00000000000000000000000000000000 P bacae03153b74596ba962b8db7fedc8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be929da42de644648042b1c514d26d16" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 46579 } } peers { permanent_uuid: "28f68f09769f4576ab963f9249cdf9e1" member_type: VOTER last_known_addr { host: "127.9.176.253" port: 37039 } } peers { permanent_uuid: "bacae03153b74596ba962b8db7fedc8f" member_type: VOTER last_known_addr { host: "127.9.176.252" port: 34903 } }
I20260808 22:22:19.506677 20237 sys_catalog.cc:455] T 00000000000000000000000000000000 P bacae03153b74596ba962b8db7fedc8f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be929da42de644648042b1c514d26d16" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 46579 } } peers { permanent_uuid: "28f68f09769f4576ab963f9249cdf9e1" member_type: VOTER last_known_addr { host: "127.9.176.253" port: 37039 } } peers { permanent_uuid: "bacae03153b74596ba962b8db7fedc8f" member_type: VOTER last_known_addr { host: "127.9.176.252" port: 34903 } } }
I20260808 22:22:19.506789 20237 sys_catalog.cc:458] T 00000000000000000000000000000000 P bacae03153b74596ba962b8db7fedc8f [sys.catalog]: This master's current role is: FOLLOWER
I20260808 22:22:19.507352 20228 sys_catalog.cc:564] T 00000000000000000000000000000000 P bacae03153b74596ba962b8db7fedc8f [sys.catalog]: configured and running, proceeding with master startup.
W20260808 22:22:19.512660 20252 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bacae03153b74596ba962b8db7fedc8f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 22:22:19.512727 20252 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 22:22:19.516124 19993 master_runner.cc:428] Detected that this master bacae03153b74596ba962b8db7fedc8f is joining an existing cluster
I20260808 22:22:19.516191 19993 master_runner.cc:432] Initiating AddMaster RPC to add 127.9.176.252:34903
I20260808 22:22:19.523332 19389 heartbeater.cc:503] Master 127.9.176.254:46579 was elected leader, sending a full tablet report...
I20260808 22:22:19.528460 20258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:19.519097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:46579 (local address 127.0.0.1:34132)
0808 22:22:19.519384 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:19.519399 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:19.519519 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:19.520223 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:19.520226 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:19.520577 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:19.520719 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:19.520726 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:19.521830 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:19.521833 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:19.523777 (+ 1944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:19.523786 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:19.524436 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:19.524439 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:19.524584 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:19.525078 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:19.525089 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:19.525466 (+ 377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:22:19.527216 (+ 1750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:19.527218 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:19.527220 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:19.527398 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:19.527645 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:19.527647 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:19.527648 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:19.527716 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:19.528071 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:22:19.528073 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:22:19.528143 (+ 70us) client_negotiation.cc:769] Sending connection context
0808 22:22:19.528230 (+ 87us) client_negotiation.cc:240] Negotiation successful
0808 22:22:19.528328 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":23,"thread_start_us":107,"threads_started":1}
I20260808 22:22:19.528559 19143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:19.519800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34132 (local address 127.9.176.254:46579)
0808 22:22:19.519917 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:19.519922 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:19.519935 (+ 13us) server_negotiation.cc:407] Connection header received
0808 22:22:19.519975 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:19.519977 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:19.520026 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:19.520100 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:19.520867 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:19.521697 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:19.523902 (+ 2205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:19.524230 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:19.524304 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:19.525603 (+ 1299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:19.525607 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:19.525610 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:22:19.525636 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:19.527100 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:19.527477 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:19.527481 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:19.527483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:19.527567 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:19.527782 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:19.527785 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:19.527786 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:19.527993 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:19.528089 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:19.528295 (+ 206us) server_negotiation.cc:299] Negotiation successful
0808 22:22:19.528409 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 22:22:19.530567 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 19173
Aug 08 22:22:19 dist-test-slave-ncp9 krb5kdc[19010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227738, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.176.254@KRBTEST.COM for oryx/127.9.176.252@KRBTEST.COM
I20260808 22:22:19.548234 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.193:34919
--local_ip_for_outbound_sockets=127.9.176.193
--tserver_master_addrs=127.9.176.254:46579
--webserver_port=41841
--webserver_interface=127.9.176.193
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/krb5kdc/oryx_127.9.176.193.keytab
--principal=oryx/127.9.176.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:43621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.9.176.254:46579,127.9.176.253:37039
--tserver_master_addrs=127.9.176.254:46579,127.9.176.253:37039,127.9.176.252:34903 with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 22:22:19.549638 20235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:19.519712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36066 (local address 127.9.176.253:37039)
0808 22:22:19.519844 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:19.519848 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:19.520172 (+ 324us) server_negotiation.cc:407] Connection header received
0808 22:22:19.520327 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:19.520330 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:19.520380 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:19.520432 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:19.521065 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:19.521801 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:19.523147 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:19.523503 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:19.523574 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:19.546004 (+ 22430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:19.546008 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:19.546010 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:22:19.546073 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:19.547656 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:19.548179 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:19.548183 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:19.548185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:19.548284 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:19.548612 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:19.548616 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:19.548618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:19.548838 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:19.548939 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:19.549301 (+ 362us) server_negotiation.cc:299] Negotiation successful
0808 22:22:19.549462 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 22:22:19.550033 19972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:19.534525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.252:34903 (local address 127.0.0.1:37070)
0808 22:22:19.534646 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:19.534663 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:19.534749 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:19.541246 (+ 6497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:19.541250 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:19.541453 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:19.541561 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:19.541567 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:19.542427 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:19.542431 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:19.542981 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:19.542990 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:19.543429 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:19.543433 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:19.543532 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:19.543979 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:19.543991 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:19.545607 (+ 1616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:22:19.548174 (+ 2567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:19.548177 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:19.548179 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:19.548431 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:19.549042 (+ 611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:19.549056 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:19.549058 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:19.549149 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:19.549562 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:22:19.549565 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:22:19.549666 (+ 101us) client_negotiation.cc:769] Sending connection context
0808 22:22:19.549746 (+ 80us) client_negotiation.cc:240] Negotiation successful
0808 22:22:19.549890 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260808 22:22:19.550592 20264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:19.519515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.253:37039 (local address 127.0.0.1:36066)
0808 22:22:19.520099 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:19.520112 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:19.520226 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:19.520577 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:19.520579 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:19.520751 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:19.520935 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:19.520945 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:19.521918 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:19.521922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:19.523002 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:19.523014 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:19.523800 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:19.523803 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:19.524237 (+ 434us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:19.545377 (+ 21140us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:19.545392 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:19.545819 (+ 427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:22:19.547792 (+ 1973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:19.547796 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:19.547798 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:19.548076 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:19.548402 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:19.548405 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:19.548407 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:19.548493 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:19.549010 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:22:19.549012 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:22:19.549113 (+ 101us) client_negotiation.cc:769] Sending connection context
0808 22:22:19.550026 (+ 913us) client_negotiation.cc:240] Negotiation successful
0808 22:22:19.550184 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"mutex_wait_us":47,"spinlock_wait_cycles":2944,"thread_start_us":260,"threads_started":1}
I20260808 22:22:19.552069 20265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:19.540100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37070 (local address 127.9.176.252:34903)
0808 22:22:19.540922 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:19.540930 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:19.540946 (+ 16us) server_negotiation.cc:407] Connection header received
0808 22:22:19.541024 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:19.541027 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:19.541092 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:19.541658 (+ 566us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:19.541752 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:19.542313 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:19.543098 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:19.543323 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:19.543407 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:19.545764 (+ 2357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:19.545768 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:19.545771 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:22:19.545807 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:19.548053 (+ 2246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:19.548819 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:19.548823 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:19.548824 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:19.548932 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:19.549256 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:19.549259 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:19.549260 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:19.549449 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:19.550986 (+ 1537us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:19.551085 (+ 99us) server_negotiation.cc:299] Negotiation successful
0808 22:22:19.551899 (+ 814us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":738,"spinlock_wait_cycles":6272,"thread_start_us":600,"threads_started":1}
I20260808 22:22:19.553028 19077 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.9.176.252:34903: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "bacae03153b74596ba962b8db7fedc8f"
member_type: NON_VOTER
last_known_addr {
host: "127.9.176.252"
port: 34903
}
attrs {
promote: true
}
}
dest_uuid: "be929da42de644648042b1c514d26d16"
cas_config_opid_index: 5
I20260808 22:22:19.553239 20232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:19.519485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37062 (local address 127.9.176.252:34903)
0808 22:22:19.519689 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:19.519693 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:19.519811 (+ 118us) server_negotiation.cc:407] Connection header received
0808 22:22:19.519949 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:19.519952 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:19.519998 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:19.520075 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:19.520870 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:19.521672 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:19.523103 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:19.523356 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:19.523430 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:19.550210 (+ 26780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:19.550214 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:19.550216 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:22:19.550238 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:19.551729 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:19.552100 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:19.552104 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:19.552106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:19.552190 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:19.552427 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:19.552430 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:19.552431 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:19.552636 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:19.552704 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:19.553022 (+ 318us) server_negotiation.cc:299] Negotiation successful
0808 22:22:19.553123 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139}
I20260808 22:22:19.553673 20262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:19.519157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.252:34903 (local address 127.0.0.1:37062)
0808 22:22:19.519741 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:19.519753 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:19.519864 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:19.520195 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:19.520197 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:19.520504 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:19.520740 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:19.520754 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:19.521796 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:19.521799 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:19.522985 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:19.522996 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:19.523575 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:19.523578 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:19.523668 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:19.524485 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:19.524500 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:19.550069 (+ 25569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:22:19.551826 (+ 1757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:19.551828 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:19.551830 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:19.552009 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:19.552273 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:19.552276 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:19.552277 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:19.552345 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:19.552759 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:22:19.552762 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:22:19.552844 (+ 82us) client_negotiation.cc:769] Sending connection context
0808 22:22:19.553386 (+ 542us) client_negotiation.cc:240] Negotiation successful
0808 22:22:19.553510 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"mutex_wait_us":136,"thread_start_us":74,"threads_started":1}
I20260808 22:22:19.553727 19077 consensus_queue.cc:237] T 00000000000000000000000000000000 P be929da42de644648042b1c514d26d16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "be929da42de644648042b1c514d26d16" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 46579 } } peers { permanent_uuid: "28f68f09769f4576ab963f9249cdf9e1" member_type: NON_VOTER last_known_addr { host: "127.9.176.253" port: 37039 } attrs { promote: true } } peers { permanent_uuid: "bacae03153b74596ba962b8db7fedc8f" member_type: NON_VOTER last_known_addr { host: "127.9.176.252" port: 34903 } attrs { promote: true } }
I20260808 22:22:19.554802 19974 raft_consensus.cc:2932] T 00000000000000000000000000000000 P be929da42de644648042b1c514d26d16 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER bacae03153b74596ba962b8db7fedc8f (127.9.176.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "be929da42de644648042b1c514d26d16" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 46579 } } peers { permanent_uuid: "28f68f09769f4576ab963f9249cdf9e1" member_type: NON_VOTER last_known_addr { host: "127.9.176.253" port: 37039 } attrs { promote: true } } peers { permanent_uuid: "bacae03153b74596ba962b8db7fedc8f" member_type: NON_VOTER last_known_addr { host: "127.9.176.252" port: 34903 } attrs { promote: true } } }
I20260808 22:22:19.555312 20182 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bacae03153b74596ba962b8db7fedc8f [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:22:19.556236 19974 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.9.176.252:34903
I20260808 22:22:19.556553 20182 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bacae03153b74596ba962b8db7fedc8f [term 1 FOLLOWER]: Refusing update from remote peer be929da42de644648042b1c514d26d16: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260808 22:22:19.556545 19974 sys_catalog.cc:455] T 00000000000000000000000000000000 P be929da42de644648042b1c514d26d16 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "be929da42de644648042b1c514d26d16" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "be929da42de644648042b1c514d26d16" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 46579 } } peers { permanent_uuid: "28f68f09769f4576ab963f9249cdf9e1" member_type: NON_VOTER last_known_addr { host: "127.9.176.253" port: 37039 } attrs { promote: true } } peers { permanent_uuid: "bacae03153b74596ba962b8db7fedc8f" member_type: NON_VOTER last_known_addr { host: "127.9.176.252" port: 34903 } attrs { promote: true } } }
I20260808 22:22:19.556640 19974 sys_catalog.cc:458] T 00000000000000000000000000000000 P be929da42de644648042b1c514d26d16 [sys.catalog]: This master's current role is: LEADER
I20260808 22:22:19.557153 19975 consensus_queue.cc:1035] T 00000000000000000000000000000000 P be929da42de644648042b1c514d26d16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bacae03153b74596ba962b8db7fedc8f" member_type: NON_VOTER last_known_addr { host: "127.9.176.252" port: 34903 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:22:19.557451 19993 master.cc:537] Master@127.9.176.252:34903 shutting down...
W20260808 22:22:19.559175 19056 consensus_peers.cc:487] T 00000000000000000000000000000000 P be929da42de644648042b1c514d26d16 -> Peer bacae03153b74596ba962b8db7fedc8f (127.9.176.252:34903): Couldn't send request to peer bacae03153b74596ba962b8db7fedc8f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260808 22:22:19.562299 19993 raft_consensus.cc:2229] T 00000000000000000000000000000000 P bacae03153b74596ba962b8db7fedc8f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:22:19.562536 19972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:19.557562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.253:37039 (local address 127.0.0.1:36082)
0808 22:22:19.557793 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:19.557807 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:19.557882 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:19.558514 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:19.558518 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:19.558533 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:22:19.558616 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:19.558622 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:19.559702 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:19.559705 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:19.560871 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:19.560883 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:19.562078 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:19.562082 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:19.562202 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:19.562255 (+ 53us) client_negotiation.cc:769] Sending connection context
0808 22:22:19.562336 (+ 81us) client_negotiation.cc:240] Negotiation successful
0808 22:22:19.562412 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158}
I20260808 22:22:19.562597 19993 raft_consensus.cc:2258] T 00000000000000000000000000000000 P bacae03153b74596ba962b8db7fedc8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:22:19.562714 19993 tablet_replica.cc:330] T 00000000000000000000000000000000 P bacae03153b74596ba962b8db7fedc8f: stopping tablet replica
I20260808 22:22:19.562873 20235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:19.558021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36082 (local address 127.9.176.253:37039)
0808 22:22:19.558183 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:19.558188 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:19.558201 (+ 13us) server_negotiation.cc:407] Connection header received
0808 22:22:19.558242 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:19.558245 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:19.558302 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:19.558388 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:22:19.558736 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:19.559577 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:19.561052 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:19.561714 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:19.561829 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:19.562003 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:19.562457 (+ 454us) server_negotiation.cc:299] Negotiation successful
0808 22:22:19.562540 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260808 22:22:19.563359 20053 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 28f68f09769f4576ab963f9249cdf9e1 [term 1 LEARNER]: Refusing update from remote peer be929da42de644648042b1c514d26d16: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260808 22:22:19.563812 20267 consensus_queue.cc:1035] T 00000000000000000000000000000000 P be929da42de644648042b1c514d26d16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28f68f09769f4576ab963f9249cdf9e1" member_type: NON_VOTER last_known_addr { host: "127.9.176.253" port: 37039 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260808 22:22:19.566051 20053 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 28f68f09769f4576ab963f9249cdf9e1 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER bacae03153b74596ba962b8db7fedc8f (127.9.176.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "be929da42de644648042b1c514d26d16" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 46579 } } peers { permanent_uuid: "28f68f09769f4576ab963f9249cdf9e1" member_type: NON_VOTER last_known_addr { host: "127.9.176.253" port: 37039 } attrs { promote: true } } peers { permanent_uuid: "bacae03153b74596ba962b8db7fedc8f" member_type: NON_VOTER last_known_addr { host: "127.9.176.252" port: 34903 } attrs { promote: true } } }
I20260808 22:22:19.567625 19993 master.cc:552] Master@127.9.176.252:34903 shutdown complete.
I20260808 22:22:19.567662 19993 master_runner.cc:305] Clearing existing system tablet
I20260808 22:22:19.567883 19993 fs_manager.cc:359] Metadata directory not provided
I20260808 22:22:19.567924 19993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:22:19.568478 20098 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28f68f09769f4576ab963f9249cdf9e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader be929da42de644648042b1c514d26d16. Latest consensus state: current_term: 1 leader_uuid: "be929da42de644648042b1c514d26d16" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "be929da42de644648042b1c514d26d16" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 46579 } } peers { permanent_uuid: "28f68f09769f4576ab963f9249cdf9e1" member_type: NON_VOTER last_known_addr { host: "127.9.176.253" port: 37039 } attrs { promote: true } } peers { permanent_uuid: "bacae03153b74596ba962b8db7fedc8f" member_type: NON_VOTER last_known_addr { host: "127.9.176.252" port: 34903 } attrs { promote: true } } }
I20260808 22:22:19.568605 20098 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28f68f09769f4576ab963f9249cdf9e1 [sys.catalog]: This master's current role is: LEARNER
I20260808 22:22:19.568707 20098 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28f68f09769f4576ab963f9249cdf9e1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "be929da42de644648042b1c514d26d16" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "be929da42de644648042b1c514d26d16" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 46579 } } peers { permanent_uuid: "28f68f09769f4576ab963f9249cdf9e1" member_type: NON_VOTER last_known_addr { host: "127.9.176.253" port: 37039 } attrs { promote: true } } peers { permanent_uuid: "bacae03153b74596ba962b8db7fedc8f" member_type: NON_VOTER last_known_addr { host: "127.9.176.252" port: 34903 } attrs { promote: true } } }
I20260808 22:22:19.568794 20098 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28f68f09769f4576ab963f9249cdf9e1 [sys.catalog]: This master's current role is: LEARNER
I20260808 22:22:19.569432 20267 raft_consensus.cc:1050] T 00000000000000000000000000000000 P be929da42de644648042b1c514d26d16: attempting to promote NON_VOTER 28f68f09769f4576ab963f9249cdf9e1 to VOTER
I20260808 22:22:19.569859 20267 consensus_queue.cc:237] T 00000000000000000000000000000000 P be929da42de644648042b1c514d26d16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "be929da42de644648042b1c514d26d16" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 46579 } } peers { permanent_uuid: "28f68f09769f4576ab963f9249cdf9e1" member_type: VOTER last_known_addr { host: "127.9.176.253" port: 37039 } attrs { promote: false } } peers { permanent_uuid: "bacae03153b74596ba962b8db7fedc8f" member_type: NON_VOTER last_known_addr { host: "127.9.176.252" port: 34903 } attrs { promote: true } }
I20260808 22:22:19.571187 19972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:19.570644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.252:34903 (local address 127.0.0.1:37086)
0808 22:22:19.571051 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:19.571119 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.252:34903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342}
I20260808 22:22:19.571955 19972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:19.571736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.252:34903 (local address 127.0.0.1:37102)
0808 22:22:19.571854 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:19.571896 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.252:34903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260808 22:22:19.572132 19056 consensus_peers.cc:487] T 00000000000000000000000000000000 P be929da42de644648042b1c514d26d16 -> Peer bacae03153b74596ba962b8db7fedc8f (127.9.176.252:34903): Couldn't send request to peer bacae03153b74596ba962b8db7fedc8f. Status: Network error: Client connection negotiation failed: client connection to 127.9.176.252:34903: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260808 22:22:19.572628 20053 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 28f68f09769f4576ab963f9249cdf9e1 [term 1 LEARNER]: Refusing update from remote peer be929da42de644648042b1c514d26d16: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260808 22:22:19.573117 19974 consensus_queue.cc:1035] T 00000000000000000000000000000000 P be929da42de644648042b1c514d26d16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28f68f09769f4576ab963f9249cdf9e1" member_type: VOTER last_known_addr { host: "127.9.176.253" port: 37039 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260808 22:22:19.574801 19974 raft_consensus.cc:2932] T 00000000000000000000000000000000 P be929da42de644648042b1c514d26d16 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 28f68f09769f4576ab963f9249cdf9e1 (127.9.176.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "be929da42de644648042b1c514d26d16" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 46579 } } peers { permanent_uuid: "28f68f09769f4576ab963f9249cdf9e1" member_type: VOTER last_known_addr { host: "127.9.176.253" port: 37039 } attrs { promote: false } } peers { permanent_uuid: "bacae03153b74596ba962b8db7fedc8f" member_type: NON_VOTER last_known_addr { host: "127.9.176.252" port: 34903 } attrs { promote: true } } }
I20260808 22:22:19.576084 19974 sys_catalog.cc:455] T 00000000000000000000000000000000 P be929da42de644648042b1c514d26d16 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "be929da42de644648042b1c514d26d16" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "be929da42de644648042b1c514d26d16" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 46579 } } peers { permanent_uuid: "28f68f09769f4576ab963f9249cdf9e1" member_type: VOTER last_known_addr { host: "127.9.176.253" port: 37039 } attrs { promote: false } } peers { permanent_uuid: "bacae03153b74596ba962b8db7fedc8f" member_type: NON_VOTER last_known_addr { host: "127.9.176.252" port: 34903 } attrs { promote: true } } }
I20260808 22:22:19.576176 19974 sys_catalog.cc:458] T 00000000000000000000000000000000 P be929da42de644648042b1c514d26d16 [sys.catalog]: This master's current role is: LEADER
I20260808 22:22:19.576292 19974 sys_catalog.cc:455] T 00000000000000000000000000000000 P be929da42de644648042b1c514d26d16 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "be929da42de644648042b1c514d26d16" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "be929da42de644648042b1c514d26d16" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 46579 } } peers { permanent_uuid: "28f68f09769f4576ab963f9249cdf9e1" member_type: VOTER last_known_addr { host: "127.9.176.253" port: 37039 } attrs { promote: false } } peers { permanent_uuid: "bacae03153b74596ba962b8db7fedc8f" member_type: NON_VOTER last_known_addr { host: "127.9.176.252" port: 34903 } attrs { promote: true } } }
I20260808 22:22:19.576352 19974 sys_catalog.cc:458] T 00000000000000000000000000000000 P be929da42de644648042b1c514d26d16 [sys.catalog]: This master's current role is: LEADER
I20260808 22:22:19.577014 19993 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.000s sys 0.002s
I20260808 22:22:19.576970 20053 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 28f68f09769f4576ab963f9249cdf9e1 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 28f68f09769f4576ab963f9249cdf9e1 (127.9.176.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "be929da42de644648042b1c514d26d16" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 46579 } } peers { permanent_uuid: "28f68f09769f4576ab963f9249cdf9e1" member_type: VOTER last_known_addr { host: "127.9.176.253" port: 37039 } attrs { promote: false } } peers { permanent_uuid: "bacae03153b74596ba962b8db7fedc8f" member_type: NON_VOTER last_known_addr { host: "127.9.176.252" port: 34903 } attrs { promote: true } } }
I20260808 22:22:19.577910 20274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:19.578121 19993 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:22:19.578231 19993 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 22:22:19.578261 19993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/master-2/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/master-2/wal
uuid: "bacae03153b74596ba962b8db7fedc8f"
format_stamp: "Formatted at 2026-08-08 22:22:19 on dist-test-slave-ncp9"
server_key: "3602e939c96672bd7fcd1765d8676dd2"
server_key_iv: "93fa90579f4157de3088f4e139613fba"
server_key_version: "encryptionkey@0"
I20260808 22:22:19.578418 20272 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28f68f09769f4576ab963f9249cdf9e1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "be929da42de644648042b1c514d26d16" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "be929da42de644648042b1c514d26d16" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 46579 } } peers { permanent_uuid: "28f68f09769f4576ab963f9249cdf9e1" member_type: VOTER last_known_addr { host: "127.9.176.253" port: 37039 } attrs { promote: false } } peers { permanent_uuid: "bacae03153b74596ba962b8db7fedc8f" member_type: NON_VOTER last_known_addr { host: "127.9.176.252" port: 34903 } attrs { promote: true } } }
I20260808 22:22:19.578523 20272 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28f68f09769f4576ab963f9249cdf9e1 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 22:22:19.580324 19993 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P bacae03153b74596ba962b8db7fedc8f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:22:19.582595 19993 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P bacae03153b74596ba962b8db7fedc8f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260808 22:22:19.582687 19993 log.cc:1196] T 00000000000000000000000000000000 P bacae03153b74596ba962b8db7fedc8f: Deleting WAL directory at /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260808 22:22:19.582922 19993 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P bacae03153b74596ba962b8db7fedc8f: Deleting consensus metadata
I20260808 22:22:19.583081 19993 master_runner.cc:315] Copying system tablet from 127.9.176.254:46579
I20260808 22:22:19.584194 19993 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P bacae03153b74596ba962b8db7fedc8f: tablet copy: Beginning tablet copy session from remote peer at address 127.9.176.254:46579
I20260808 22:22:19.593367 19143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:19.585236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34144 (local address 127.9.176.254:46579)
0808 22:22:19.585359 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:19.585364 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:19.585588 (+ 224us) server_negotiation.cc:407] Connection header received
0808 22:22:19.585711 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:19.585714 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:19.585763 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:19.585836 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:19.586305 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:19.587095 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:19.588262 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:19.588579 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:19.588691 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:19.589966 (+ 1275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:19.589970 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:19.589973 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:22:19.589999 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:19.591554 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:19.592085 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:19.592089 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:19.592091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:19.592190 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:19.592464 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:19.592467 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:19.592469 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:19.592694 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:19.592789 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:19.593093 (+ 304us) server_negotiation.cc:299] Negotiation successful
0808 22:22:19.593213 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260808 22:22:19.593412 20280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:19.585175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:46579 (local address 127.0.0.1:34144)
0808 22:22:19.585483 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:19.585496 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:19.585618 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:19.585862 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:19.585865 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:19.586072 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:19.586183 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:19.586191 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:19.587213 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:19.587216 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:19.588141 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:19.588152 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:19.588687 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:19.588691 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:19.588775 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:19.589376 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:19.589390 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:19.589811 (+ 421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:22:19.591667 (+ 1856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:19.591671 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:19.591673 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:19.591985 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:19.592277 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:19.592280 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:19.592282 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:19.592376 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:19.592782 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:22:19.592785 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:22:19.592884 (+ 99us) client_negotiation.cc:769] Sending connection context
0808 22:22:19.593020 (+ 136us) client_negotiation.cc:240] Negotiation successful
0808 22:22:19.593217 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":90,"threads_started":1}
I20260808 22:22:19.594059 19112 tablet_copy_service.cc:140] P be929da42de644648042b1c514d26d16: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer bacae03153b74596ba962b8db7fedc8f ({username='oryx', principal='oryx/127.9.176.252@KRBTEST.COM'} at 127.0.0.1:34144)
I20260808 22:22:19.594161 19112 tablet_copy_service.cc:161] P be929da42de644648042b1c514d26d16: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer bacae03153b74596ba962b8db7fedc8f at {username='oryx', principal='oryx/127.9.176.252@KRBTEST.COM'} at 127.0.0.1:34144: session id = bacae03153b74596ba962b8db7fedc8f-00000000000000000000000000000000
I20260808 22:22:19.595221 19112 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P be929da42de644648042b1c514d26d16: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 22:22:19.596681 19993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260808 22:22:19.598995 19993 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P bacae03153b74596ba962b8db7fedc8f: tablet copy: Starting download of 0 data blocks...
I20260808 22:22:19.599185 19993 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P bacae03153b74596ba962b8db7fedc8f: tablet copy: Starting download of 1 WAL segments...
I20260808 22:22:19.600634 19993 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P bacae03153b74596ba962b8db7fedc8f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 22:22:19.602355 19112 tablet_copy_service.cc:342] P be929da42de644648042b1c514d26d16: Request end of tablet copy session bacae03153b74596ba962b8db7fedc8f-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.9.176.252@KRBTEST.COM'} at 127.0.0.1:34144
I20260808 22:22:19.602437 19112 tablet_copy_service.cc:433] P be929da42de644648042b1c514d26d16: ending tablet copy session bacae03153b74596ba962b8db7fedc8f-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer bacae03153b74596ba962b8db7fedc8f
I20260808 22:22:19.609863 19993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:22:19.611567 20281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:19.611802 20282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:19.611928 20285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:22:19.612388 19993 server_base.cc:1034] running on GCE node
I20260808 22:22:19.612541 19993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:22:19.612834 19993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:22:19.613981 19993 hybrid_clock.cc:648] HybridClock initialized: now 1786227739613961 us; error 36 us; skew 500 ppm
Aug 08 22:22:19 dist-test-slave-ncp9 krb5kdc[19010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227739, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.176.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:22:19.615625 19993 init.cc:377] Logged in from keytab as oryx/127.9.176.252@KRBTEST.COM (short username oryx)
I20260808 22:22:19.616147 19993 webserver.cc:460] Webserver started at http://127.9.176.252:42105/ using document root <none> and password file <none>
I20260808 22:22:19.616348 19993 fs_manager.cc:359] Metadata directory not provided
I20260808 22:22:19.616396 19993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:22:19.617849 19993 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 22:22:19.618911 20300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:19.619166 19993 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 22:22:19.619290 19993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/master-2/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/master-2/wal
uuid: "bacae03153b74596ba962b8db7fedc8f"
format_stamp: "Formatted at 2026-08-08 22:22:19 on dist-test-slave-ncp9"
server_key: "3602e939c96672bd7fcd1765d8676dd2"
server_key_iv: "93fa90579f4157de3088f4e139613fba"
server_key_version: "encryptionkey@0"
I20260808 22:22:19.619449 19993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:22:19.660856 19993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:22:19.663340 19993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:22:19.669322 19993 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.252:34903
I20260808 22:22:19.669606 20381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.252:34903 every 8 connection(s)
I20260808 22:22:19.670207 19993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/master-2/data/info.pb
I20260808 22:22:19.672225 20382 sys_catalog.cc:263] Verifying existing consensus state
I20260808 22:22:19.673305 20382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bacae03153b74596ba962b8db7fedc8f: Bootstrap starting.
I20260808 22:22:19.685134 20382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bacae03153b74596ba962b8db7fedc8f: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 22:22:19.685451 20382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bacae03153b74596ba962b8db7fedc8f: Bootstrap complete.
I20260808 22:22:19.686053 20382 raft_consensus.cc:348] T 00000000000000000000000000000000 P bacae03153b74596ba962b8db7fedc8f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "be929da42de644648042b1c514d26d16" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 46579 } } peers { permanent_uuid: "28f68f09769f4576ab963f9249cdf9e1" member_type: VOTER last_known_addr { host: "127.9.176.253" port: 37039 } attrs { promote: false } } peers { permanent_uuid: "bacae03153b74596ba962b8db7fedc8f" member_type: NON_VOTER last_known_addr { host: "127.9.176.252" port: 34903 } attrs { promote: true } }
I20260808 22:22:19.686213 20382 raft_consensus.cc:729] T 00000000000000000000000000000000 P bacae03153b74596ba962b8db7fedc8f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bacae03153b74596ba962b8db7fedc8f, State: Initialized, Role: LEARNER
I20260808 22:22:19.686347 20382 consensus_queue.cc:260] T 00000000000000000000000000000000 P bacae03153b74596ba962b8db7fedc8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "be929da42de644648042b1c514d26d16" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 46579 } } peers { permanent_uuid: "28f68f09769f4576ab963f9249cdf9e1" member_type: VOTER last_known_addr { host: "127.9.176.253" port: 37039 } attrs { promote: false } } peers { permanent_uuid: "bacae03153b74596ba962b8db7fedc8f" member_type: NON_VOTER last_known_addr { host: "127.9.176.252" port: 34903 } attrs { promote: true } }
I20260808 22:22:19.686708 20386 sys_catalog.cc:455] T 00000000000000000000000000000000 P bacae03153b74596ba962b8db7fedc8f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "be929da42de644648042b1c514d26d16" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 46579 } } peers { permanent_uuid: "28f68f09769f4576ab963f9249cdf9e1" member_type: VOTER last_known_addr { host: "127.9.176.253" port: 37039 } attrs { promote: false } } peers { permanent_uuid: "bacae03153b74596ba962b8db7fedc8f" member_type: NON_VOTER last_known_addr { host: "127.9.176.252" port: 34903 } attrs { promote: true } } }
I20260808 22:22:19.686812 20386 sys_catalog.cc:458] T 00000000000000000000000000000000 P bacae03153b74596ba962b8db7fedc8f [sys.catalog]: This master's current role is: LEARNER
I20260808 22:22:19.686878 20382 sys_catalog.cc:564] T 00000000000000000000000000000000 P bacae03153b74596ba962b8db7fedc8f [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:22:19.689854 20404 catalog_manager.cc:1259] Loaded cluster ID: 68a7dc5d6ee2417c9d58328afa544c53
I20260808 22:22:19.689956 20404 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bacae03153b74596ba962b8db7fedc8f: loading cluster ID for follower catalog manager: success
I20260808 22:22:19.691697 20404 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bacae03153b74596ba962b8db7fedc8f: acquiring CA information for follower catalog manager: success
I20260808 22:22:19.692760 20404 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bacae03153b74596ba962b8db7fedc8f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260808 22:22:19.702764 20266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:22:19.702989 20266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:22:19.703049 20266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:22:19.707063 20266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:22:19.707118 20266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:22:19.707212 20266 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.193
I20260808 22:22:19.712343 20266 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:43621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/krb5kdc/oryx_127.9.176.193.keytab
--principal=oryx/127.9.176.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.193:34919
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.176.193
--webserver_port=41841
--tserver_master_addrs=127.9.176.254:46579,127.9.176.253:37039,127.9.176.252:34903
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20266
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.193
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:22:19.713404 20266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:22:19.714377 20266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:22:19.724236 20414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:19.724280 20413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:22:19.724548 20266 server_base.cc:1034] running on GCE node
W20260808 22:22:19.724224 20417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:22:19.724996 20266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:22:19.725594 20266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:22:19.726750 20266 hybrid_clock.cc:648] HybridClock initialized: now 1786227739726716 us; error 41 us; skew 500 ppm
Aug 08 22:22:19 dist-test-slave-ncp9 krb5kdc[19010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227739, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.176.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:22:19.729465 20266 init.cc:377] Logged in from keytab as oryx/127.9.176.193@KRBTEST.COM (short username oryx)
I20260808 22:22:19.730278 20266 webserver.cc:460] Webserver started at http://127.9.176.193:41841/ using document root <none> and password file <none>
I20260808 22:22:19.730767 20266 fs_manager.cc:359] Metadata directory not provided
I20260808 22:22:19.730840 20266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:22:19.734560 20266 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260808 22:22:19.736445 20425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:19.737289 19606 heartbeater.cc:503] Master 127.9.176.254:46579 was elected leader, sending a full tablet report...
I20260808 22:22:19.737761 20266 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 22:22:19.737879 20266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-0/wal
uuid: "39101d245faa4b6eb281ffc89ef69bf6"
format_stamp: "Formatted at 2026-08-08 22:22:18 on dist-test-slave-ncp9"
server_key: "c9342399e589036eda2af070e7c2a46d"
server_key_iv: "a5ea9178152bf8f6cb6bcb27f9edd517"
server_key_version: "encryptionkey@0"
I20260808 22:22:19.738404 20266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:22:19.784376 20266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:22:19.787416 20266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:22:19.787611 20266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:22:19.788314 20266 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:22:19.789260 20266 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:22:19.789307 20266 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:19.789350 20266 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:22:19.789377 20266 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:19.801270 20266 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.193:34919
I20260808 22:22:19.801290 20613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.193:34919 every 8 connection(s)
I20260808 22:22:19.802253 20266 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
I20260808 22:22:19.802542 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 20266
I20260808 22:22:19.802731 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 19395
Aug 08 22:22:19 dist-test-slave-ncp9 krb5kdc[19010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227739, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.176.193@KRBTEST.COM for oryx/127.9.176.252@KRBTEST.COM
Aug 08 22:22:19 dist-test-slave-ncp9 krb5kdc[19010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227739, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.176.193@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM
Aug 08 22:22:19 dist-test-slave-ncp9 krb5kdc[19010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227739, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.176.193@KRBTEST.COM for oryx/127.9.176.253@KRBTEST.COM
I20260808 22:22:19.820375 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.194:46339
--local_ip_for_outbound_sockets=127.9.176.194
--tserver_master_addrs=127.9.176.254:46579
--webserver_port=45029
--webserver_interface=127.9.176.194
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/krb5kdc/oryx_127.9.176.194.keytab
--principal=oryx/127.9.176.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:43621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.9.176.254:46579,127.9.176.253:37039
--tserver_master_addrs=127.9.176.254:46579,127.9.176.253:37039,127.9.176.252:34903 with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 22:22:19.822781 20618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:19.805935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.193:42251 (local address 127.9.176.252:34903)
0808 22:22:19.807974 (+ 2039us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:19.807978 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:19.808165 (+ 187us) server_negotiation.cc:407] Connection header received
0808 22:22:19.808311 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:19.808314 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:19.808358 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:19.808407 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:19.809859 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:19.810789 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:19.812237 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:19.812793 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:19.812954 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:19.816361 (+ 3407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:19.816365 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:19.816367 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:22:19.816394 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:19.819707 (+ 3313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:19.820424 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:19.820427 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:19.820429 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:19.820761 (+ 332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:19.821205 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:19.821209 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:19.821210 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:19.821424 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:19.821515 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:19.822481 (+ 966us) server_negotiation.cc:299] Negotiation successful
0808 22:22:19.822606 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1967,"thread_start_us":81,"threads_started":1}
I20260808 22:22:19.824085 20235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:19.814582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.193:41391 (local address 127.9.176.253:37039)
0808 22:22:19.814692 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:19.814696 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:19.815170 (+ 474us) server_negotiation.cc:407] Connection header received
0808 22:22:19.815327 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:19.815330 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:19.815377 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:19.815422 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:19.816029 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:19.817099 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:19.818025 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:19.818249 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:19.818337 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:19.820532 (+ 2195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:19.820536 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:19.820539 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:22:19.820565 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:19.822089 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:19.822581 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:19.822584 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:19.822586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:19.822671 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:19.823202 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:19.823206 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:19.823207 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:19.823419 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:19.823505 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:19.823838 (+ 333us) server_negotiation.cc:299] Negotiation successful
0808 22:22:19.823950 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260808 22:22:19.824113 20619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:19.805655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.252:34903 (local address 127.9.176.193:42251)
0808 22:22:19.808094 (+ 2439us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:19.808106 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:19.808223 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:19.808684 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:19.808691 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:19.809133 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:19.809697 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:19.809715 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:19.810934 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:19.810937 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:19.812099 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:19.812109 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:19.813164 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:19.813172 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:19.813611 (+ 439us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:19.814235 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:19.814255 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:19.816172 (+ 1917us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:22:19.819854 (+ 3682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:19.819861 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:19.819876 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:19.820306 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:19.820872 (+ 566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:19.820875 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:19.820877 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:19.821087 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:19.821575 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:22:19.821583 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:22:19.821888 (+ 305us) client_negotiation.cc:769] Sending connection context
0808 22:22:19.822098 (+ 210us) client_negotiation.cc:240] Negotiation successful
0808 22:22:19.823053 (+ 955us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2375,"thread_start_us":107,"threads_started":1}
I20260808 22:22:19.824812 20617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:19.804532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:46579 (local address 127.9.176.193:45983)
0808 22:22:19.806714 (+ 2182us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:19.806766 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:19.807611 (+ 845us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:19.814181 (+ 6570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:19.814185 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:19.814724 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:19.814842 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:19.814849 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:19.815921 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:19.815924 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:19.816721 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:19.816729 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:19.817252 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:19.817256 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:19.817354 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:19.817973 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:19.817986 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:19.820126 (+ 2140us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:22:19.822563 (+ 2437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:19.822566 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:19.822568 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:19.822790 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:19.823374 (+ 584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:19.823377 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:19.823379 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:19.823470 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:19.823940 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:22:19.823942 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:22:19.824024 (+ 82us) client_negotiation.cc:769] Sending connection context
0808 22:22:19.824125 (+ 101us) client_negotiation.cc:240] Negotiation successful
0808 22:22:19.824246 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1980,"thread_start_us":1425,"threads_started":1}
I20260808 22:22:19.825366 20623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:19.814239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.253:37039 (local address 127.9.176.193:41391)
0808 22:22:19.815091 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:19.815104 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:19.815233 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:19.815561 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:19.815564 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:19.815750 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:19.815892 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:19.815899 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:19.817210 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:19.817213 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:19.817873 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:19.817884 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:19.818366 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:19.818370 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:19.818458 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:19.818876 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:19.818891 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:19.820389 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:22:19.822208 (+ 1819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:19.822211 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:19.822213 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:19.822459 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:19.822779 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:19.822782 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:19.822784 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:19.823094 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:19.823578 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:22:19.823581 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:22:19.823666 (+ 85us) client_negotiation.cc:769] Sending connection context
0808 22:22:19.824670 (+ 1004us) client_negotiation.cc:240] Negotiation successful
0808 22:22:19.824806 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":793,"thread_start_us":275,"threads_started":1}
I20260808 22:22:19.825486 20614 heartbeater.cc:347] Connected to a master server at 127.9.176.252:34903
I20260808 22:22:19.825645 19143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:19.804886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.193:45983 (local address 127.9.176.254:46579)
0808 22:22:19.805024 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:19.805028 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:19.813541 (+ 8513us) server_negotiation.cc:407] Connection header received
0808 22:22:19.813608 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:19.813611 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:19.813665 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:19.813741 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:19.814994 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:19.815788 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:19.816833 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:19.817135 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:19.817235 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:19.820329 (+ 3094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:19.820332 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:19.820335 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:22:19.820363 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:19.822427 (+ 2064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:19.823158 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:19.823161 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:19.823163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:19.823259 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:19.823584 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:19.823587 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:19.823589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:19.823806 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:19.823888 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:19.825324 (+ 1436us) server_negotiation.cc:299] Negotiation successful
0808 22:22:19.825486 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260808 22:22:19.825798 20614 heartbeater.cc:464] Registering TS with master...
I20260808 22:22:19.826359 20614 heartbeater.cc:511] Master 127.9.176.252:34903 requested a full tablet report, sending...
I20260808 22:22:19.826463 20621 heartbeater.cc:347] Connected to a master server at 127.9.176.253:37039
I20260808 22:22:19.826571 20621 heartbeater.cc:464] Registering TS with master...
I20260808 22:22:19.826738 20621 heartbeater.cc:511] Master 127.9.176.253:37039 requested a full tablet report, sending...
I20260808 22:22:19.826889 20616 heartbeater.cc:347] Connected to a master server at 127.9.176.254:46579
I20260808 22:22:19.826965 20616 heartbeater.cc:464] Registering TS with master...
I20260808 22:22:19.827129 20616 heartbeater.cc:511] Master 127.9.176.254:46579 requested a full tablet report, sending...
I20260808 22:22:19.827987 20035 ts_manager.cc:194] Registered new tserver with Master: 39101d245faa4b6eb281ffc89ef69bf6 (127.9.176.193:34919)
I20260808 22:22:19.828104 20329 ts_manager.cc:194] Registered new tserver with Master: 39101d245faa4b6eb281ffc89ef69bf6 (127.9.176.193:34919)
I20260808 22:22:19.828789 19077 ts_manager.cc:194] Re-registered known tserver with Master: 39101d245faa4b6eb281ffc89ef69bf6 (127.9.176.193:34919)
I20260808 22:22:19.829602 19077 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.176.193@KRBTEST.COM'} at 127.9.176.193:45983
W20260808 22:22:19.934231 20625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:22:19.934458 20625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:22:19.934499 20625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:22:19.938354 20625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:22:19.938401 20625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:22:19.938476 20625 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.194
I20260808 22:22:19.941061 19809 heartbeater.cc:503] Master 127.9.176.254:46579 was elected leader, sending a full tablet report...
I20260808 22:22:19.943799 20625 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:43621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/krb5kdc/oryx_127.9.176.194.keytab
--principal=oryx/127.9.176.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.194:46339
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.176.194
--webserver_port=45029
--tserver_master_addrs=127.9.176.254:46579,127.9.176.253:37039,127.9.176.252:34903
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20625
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.194
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:22:19.944730 20625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:22:19.945719 20625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:22:19.955293 20631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:19.955294 20634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:22:19.955314 20625 server_base.cc:1034] running on GCE node
W20260808 22:22:19.955325 20630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:22:19.955938 20625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:22:19.956516 20625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:22:19.957675 20625 hybrid_clock.cc:648] HybridClock initialized: now 1786227739957649 us; error 40 us; skew 500 ppm
Aug 08 22:22:19 dist-test-slave-ncp9 krb5kdc[19010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227739, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.176.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:22:19.960276 20625 init.cc:377] Logged in from keytab as oryx/127.9.176.194@KRBTEST.COM (short username oryx)
I20260808 22:22:19.961314 20625 webserver.cc:460] Webserver started at http://127.9.176.194:45029/ using document root <none> and password file <none>
I20260808 22:22:19.961805 20625 fs_manager.cc:359] Metadata directory not provided
I20260808 22:22:19.961876 20625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:22:19.965576 20625 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260808 22:22:19.967641 20641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:19.968767 20625 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 22:22:19.968871 20625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-1/wal
uuid: "41b49dbbd6a04fce85c78e62b80ba81e"
format_stamp: "Formatted at 2026-08-08 22:22:18 on dist-test-slave-ncp9"
server_key: "0dba81c13d72389afd57459a60df36a5"
server_key_iv: "7af60a186758cc14fe6740c8446dc662"
server_key_version: "encryptionkey@0"
I20260808 22:22:19.969409 20625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:22:19.983789 20625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:22:19.986261 20625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:22:19.986421 20625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:22:19.986900 20625 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:22:19.987725 20625 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:22:19.987774 20625 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:19.987824 20625 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:22:19.987855 20625 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:19.995939 20625 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.194:46339
I20260808 22:22:19.995988 20817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.194:46339 every 8 connection(s)
I20260808 22:22:19.996934 20625 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
I20260808 22:22:19.998126 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 20625
I20260808 22:22:19.998291 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 19611
I20260808 22:22:20.016103 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.195:37359
--local_ip_for_outbound_sockets=127.9.176.195
--tserver_master_addrs=127.9.176.254:46579
--webserver_port=36331
--webserver_interface=127.9.176.195
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/krb5kdc/oryx_127.9.176.195.keytab
--principal=oryx/127.9.176.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:43621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.9.176.254:46579,127.9.176.253:37039
--tserver_master_addrs=127.9.176.254:46579,127.9.176.253:37039,127.9.176.252:34903 with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 22:22:20 dist-test-slave-ncp9 krb5kdc[19010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227739, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.176.194@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM
Aug 08 22:22:20 dist-test-slave-ncp9 krb5kdc[19010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227739, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.176.194@KRBTEST.COM for oryx/127.9.176.252@KRBTEST.COM
Aug 08 22:22:20 dist-test-slave-ncp9 krb5kdc[19010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227739, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.176.194@KRBTEST.COM for oryx/127.9.176.253@KRBTEST.COM
I20260808 22:22:20.023816 19143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:20.009764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.194:35389 (local address 127.9.176.254:46579)
0808 22:22:20.009890 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:20.009894 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:20.010722 (+ 828us) server_negotiation.cc:407] Connection header received
0808 22:22:20.010918 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:20.010921 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:20.010976 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:20.011026 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:20.012137 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:20.012981 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:20.014834 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:20.015173 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:20.015246 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:20.019577 (+ 4331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:20.019581 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:20.019584 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:22:20.019611 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:20.021873 (+ 2262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:20.022495 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:20.022498 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:20.022500 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:20.022585 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:20.022883 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:20.022887 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:20.022888 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:20.023103 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:20.023175 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:20.023512 (+ 337us) server_negotiation.cc:299] Negotiation successful
0808 22:22:20.023644 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 22:22:20.023816 20618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:20.009371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.194:50169 (local address 127.9.176.252:34903)
0808 22:22:20.009875 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:20.009879 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:20.009893 (+ 14us) server_negotiation.cc:407] Connection header received
0808 22:22:20.009935 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:20.009950 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:20.009994 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:20.010066 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:20.012134 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:20.012929 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:20.014828 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:20.015203 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:20.015280 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:20.019565 (+ 4285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:20.019569 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:20.019572 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:22:20.019599 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:20.021344 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:20.021976 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:20.021979 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:20.021981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:20.022072 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:20.022494 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:20.022498 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:20.022500 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:20.022742 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:20.022811 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:20.023512 (+ 701us) server_negotiation.cc:299] Negotiation successful
0808 22:22:20.023647 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413}
I20260808 22:22:20.023844 20235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:20.009223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.194:47079 (local address 127.9.176.253:37039)
0808 22:22:20.009501 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:20.009505 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:20.009521 (+ 16us) server_negotiation.cc:407] Connection header received
0808 22:22:20.009565 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:20.009568 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:20.009614 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:20.009692 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:20.012141 (+ 2449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:20.012984 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:20.014834 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:20.015176 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:20.015246 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:20.019373 (+ 4127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:20.019376 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:20.019379 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:22:20.019407 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:20.021668 (+ 2261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:20.022100 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:20.022103 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:20.022105 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:20.022219 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:20.022562 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:20.022565 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:20.022567 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:20.022796 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:20.022873 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:20.023527 (+ 654us) server_negotiation.cc:299] Negotiation successful
0808 22:22:20.023655 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199}
I20260808 22:22:20.025303 20832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:19.999168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.252:34903 (local address 127.9.176.194:50169)
0808 22:22:20.005550 (+ 6382us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:20.005632 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:20.006864 (+ 1232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:20.010769 (+ 3905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:20.010772 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:20.011150 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:20.011966 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:20.011986 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:20.013115 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:20.013118 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:20.014680 (+ 1562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:20.014691 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:20.015449 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:20.015453 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:20.015721 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:20.016582 (+ 861us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:20.016599 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:20.019346 (+ 2747us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:22:20.021487 (+ 2141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:20.021494 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:20.021512 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:20.021862 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:20.022205 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:20.022208 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:20.022209 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:20.022382 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:20.022885 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:22:20.022892 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:22:20.023219 (+ 327us) client_negotiation.cc:769] Sending connection context
0808 22:22:20.023981 (+ 762us) client_negotiation.cc:240] Negotiation successful
0808 22:22:20.024379 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6101,"mutex_wait_us":274,"spinlock_wait_cycles":21120,"thread_start_us":6733,"threads_started":1}
I20260808 22:22:20.025348 20828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:19.999170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.253:37039 (local address 127.9.176.194:47079)
0808 22:22:20.007227 (+ 8057us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:20.007239 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:20.007314 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:20.010557 (+ 3243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:20.010567 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:20.011156 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:20.011966 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:20.011988 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:20.013121 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:20.013124 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:20.014694 (+ 1570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:20.014706 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:20.015448 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:20.015452 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:20.015780 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:20.016642 (+ 862us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:20.016655 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:20.019218 (+ 2563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:22:20.021787 (+ 2569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:20.021790 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:20.021792 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:20.022002 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:20.022339 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:20.022342 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:20.022344 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:20.022448 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:20.022962 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:22:20.022966 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:22:20.023219 (+ 253us) client_negotiation.cc:769] Sending connection context
0808 22:22:20.024143 (+ 924us) client_negotiation.cc:240] Negotiation successful
0808 22:22:20.024678 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7981,"mutex_wait_us":875,"spinlock_wait_cycles":84736,"thread_start_us":3037,"threads_started":1}
I20260808 22:22:20.025308 20834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:20.009654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:46579 (local address 127.9.176.194:35389)
0808 22:22:20.010627 (+ 973us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:20.010641 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:20.010791 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:20.011171 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:20.011173 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:20.011335 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:20.011966 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:20.011988 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:20.013121 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:20.013124 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:20.014694 (+ 1570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:20.014706 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:20.015448 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:20.015452 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:20.015723 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:20.016505 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:20.016526 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:20.019424 (+ 2898us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:22:20.022137 (+ 2713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:20.022139 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:20.022141 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:20.022382 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:20.022698 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:20.022701 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:20.022702 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:20.022781 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:20.023244 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:22:20.023246 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:22:20.023326 (+ 80us) client_negotiation.cc:769] Sending connection context
0808 22:22:20.024014 (+ 688us) client_negotiation.cc:240] Negotiation successful
0808 22:22:20.024452 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":910,"mutex_wait_us":575,"spinlock_wait_cycles":5632,"thread_start_us":86,"threads_started":1}
I20260808 22:22:20.026662 20826 heartbeater.cc:347] Connected to a master server at 127.9.176.253:37039
I20260808 22:22:20.026683 20820 heartbeater.cc:347] Connected to a master server at 127.9.176.252:34903
I20260808 22:22:20.026912 20820 heartbeater.cc:464] Registering TS with master...
I20260808 22:22:20.026912 20826 heartbeater.cc:464] Registering TS with master...
I20260808 22:22:20.027256 20822 heartbeater.cc:347] Connected to a master server at 127.9.176.254:46579
I20260808 22:22:20.027326 20822 heartbeater.cc:464] Registering TS with master...
I20260808 22:22:20.027551 20822 heartbeater.cc:511] Master 127.9.176.254:46579 requested a full tablet report, sending...
I20260808 22:22:20.027562 20820 heartbeater.cc:511] Master 127.9.176.252:34903 requested a full tablet report, sending...
I20260808 22:22:20.027576 20826 heartbeater.cc:511] Master 127.9.176.253:37039 requested a full tablet report, sending...
I20260808 22:22:20.028591 19077 ts_manager.cc:194] Re-registered known tserver with Master: 41b49dbbd6a04fce85c78e62b80ba81e (127.9.176.194:46339)
I20260808 22:22:20.028671 20329 ts_manager.cc:194] Registered new tserver with Master: 41b49dbbd6a04fce85c78e62b80ba81e (127.9.176.194:46339)
I20260808 22:22:20.029332 19077 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.176.194@KRBTEST.COM'} at 127.9.176.194:35389
I20260808 22:22:20.029811 20035 ts_manager.cc:194] Registered new tserver with Master: 41b49dbbd6a04fce85c78e62b80ba81e (127.9.176.194:46339)
I20260808 22:22:20.063122 19972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:20.059486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.252:34903 (local address 127.0.0.1:37104)
0808 22:22:20.059633 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:20.059647 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:20.059731 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:20.060022 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:20.060024 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:20.060036 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:22:20.060125 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:20.060129 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:20.061172 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:20.061175 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:20.061914 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:20.061924 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:20.062814 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:20.062816 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:20.062880 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:20.062901 (+ 21us) client_negotiation.cc:769] Sending connection context
0808 22:22:20.062944 (+ 43us) client_negotiation.cc:240] Negotiation successful
0808 22:22:20.063006 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260808 22:22:20.063265 20618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:20.059585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37104 (local address 127.9.176.252:34903)
0808 22:22:20.059713 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:20.059718 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:20.059735 (+ 17us) server_negotiation.cc:407] Connection header received
0808 22:22:20.059838 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:20.059841 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:20.059903 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:20.060016 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:22:20.060213 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:20.061055 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:20.062041 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:20.062702 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:20.062823 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:20.063015 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:20.063058 (+ 43us) server_negotiation.cc:299] Negotiation successful
0808 22:22:20.063133 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 22:22:20.063872 20345 raft_consensus.cc:1203] T 00000000000000000000000000000000 P bacae03153b74596ba962b8db7fedc8f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260808 22:22:20.064227 20386 sys_catalog.cc:455] T 00000000000000000000000000000000 P bacae03153b74596ba962b8db7fedc8f [sys.catalog]: SysCatalogTable state changed. Reason: New leader be929da42de644648042b1c514d26d16. Latest consensus state: current_term: 1 leader_uuid: "be929da42de644648042b1c514d26d16" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "be929da42de644648042b1c514d26d16" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 46579 } } peers { permanent_uuid: "28f68f09769f4576ab963f9249cdf9e1" member_type: VOTER last_known_addr { host: "127.9.176.253" port: 37039 } attrs { promote: false } } peers { permanent_uuid: "bacae03153b74596ba962b8db7fedc8f" member_type: NON_VOTER last_known_addr { host: "127.9.176.252" port: 34903 } attrs { promote: true } } }
I20260808 22:22:20.064366 20386 sys_catalog.cc:458] T 00000000000000000000000000000000 P bacae03153b74596ba962b8db7fedc8f [sys.catalog]: This master's current role is: LEARNER
W20260808 22:22:20.131744 20835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:22:20.131981 20835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:22:20.132038 20835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:22:20.135931 20835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:22:20.135986 20835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:22:20.136065 20835 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.195
I20260808 22:22:20.141113 20835 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:43621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/krb5kdc/oryx_127.9.176.195.keytab
--principal=oryx/127.9.176.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.195:37359
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.176.195
--webserver_port=36331
--tserver_master_addrs=127.9.176.254:46579,127.9.176.253:37039,127.9.176.252:34903
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20835
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.195
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:22:20.142108 20835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:22:20.143100 20835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:22:20.152498 20846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:20.152529 20841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:22:20.152611 20835 server_base.cc:1034] running on GCE node
W20260808 22:22:20.152499 20842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:22:20.153172 20835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:22:20.153728 20835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:22:20.154887 20835 hybrid_clock.cc:648] HybridClock initialized: now 1786227740154869 us; error 28 us; skew 500 ppm
Aug 08 22:22:20 dist-test-slave-ncp9 krb5kdc[19010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227740, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.176.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:22:20.157557 20835 init.cc:377] Logged in from keytab as oryx/127.9.176.195@KRBTEST.COM (short username oryx)
I20260808 22:22:20.158322 20835 webserver.cc:460] Webserver started at http://127.9.176.195:36331/ using document root <none> and password file <none>
I20260808 22:22:20.158811 20835 fs_manager.cc:359] Metadata directory not provided
I20260808 22:22:20.158877 20835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:22:20.162602 20835 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260808 22:22:20.164512 20863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:20.165766 20835 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260808 22:22:20.165871 20835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-2/wal
uuid: "cf24739aa12a410fa19526e69e0c2d06"
format_stamp: "Formatted at 2026-08-08 22:22:18 on dist-test-slave-ncp9"
server_key: "b4f019a48a02e642252613ece5673125"
server_key_iv: "261c00c609d412364e5e228aec58cfeb"
server_key_version: "encryptionkey@0"
I20260808 22:22:20.166369 20835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:22:20.175275 20835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:22:20.178128 20835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:22:20.178284 20835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:22:20.178805 20835 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:22:20.179607 20835 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:22:20.179653 20835 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:20.179702 20835 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:22:20.179728 20835 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:20.189728 20835 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.195:37359
I20260808 22:22:20.189749 21030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.195:37359 every 8 connection(s)
I20260808 22:22:20.190642 20835 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
I20260808 22:22:20.192790 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 20835
Aug 08 22:22:20 dist-test-slave-ncp9 krb5kdc[19010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227740, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.176.195@KRBTEST.COM for oryx/127.9.176.253@KRBTEST.COM
Aug 08 22:22:20 dist-test-slave-ncp9 krb5kdc[19010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227740, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.176.195@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM
I20260808 22:22:20.205997 21050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:20.194770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37114 (local address 127.9.176.252:34903)
0808 22:22:20.195832 (+ 1062us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:20.195837 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:20.195850 (+ 13us) server_negotiation.cc:407] Connection header received
0808 22:22:20.196012 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:20.196015 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:20.196062 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:20.196147 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:20.197145 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:20.197831 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:20.198991 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:20.199317 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:20.199410 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:20.201739 (+ 2329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:20.201743 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:20.201746 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:22:20.201770 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:20.203529 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:20.204197 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:20.204200 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:20.204202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:20.204292 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:20.204794 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:20.204798 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:20.204799 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:20.205041 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:20.205155 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:20.205707 (+ 552us) server_negotiation.cc:299] Negotiation successful
0808 22:22:20.205821 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":995,"thread_start_us":81,"threads_started":1}
I20260808 22:22:20.208089 20235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:20.194286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.195:48487 (local address 127.9.176.253:37039)
0808 22:22:20.194401 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:20.194404 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:20.196072 (+ 1668us) server_negotiation.cc:407] Connection header received
0808 22:22:20.196257 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:20.196261 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:20.196306 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:20.196357 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:20.196882 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:20.197677 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:20.200165 (+ 2488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:20.200495 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:20.200588 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:20.203709 (+ 3121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:20.203713 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:20.203715 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:22:20.203744 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:20.205294 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:20.206351 (+ 1057us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:20.206354 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:20.206356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:20.206444 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:20.206865 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:20.206868 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:20.206870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:20.207087 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:20.207152 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:20.207825 (+ 673us) server_negotiation.cc:299] Negotiation successful
0808 22:22:20.207939 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260808 22:22:20.209018 21052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:20.195059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34150 (local address 127.9.176.254:46579)
0808 22:22:20.195295 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:20.195299 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:20.198263 (+ 2964us) server_negotiation.cc:407] Connection header received
0808 22:22:20.198303 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:20.198306 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:20.198350 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:20.198425 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:20.199300 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:20.200042 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:20.201811 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:20.202165 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:20.202283 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:20.204754 (+ 2471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:20.204758 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:20.204761 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:22:20.204787 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:20.206501 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:20.206922 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:20.206926 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:20.206928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:20.207007 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:20.207299 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:20.207302 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:20.207304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:20.207497 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:20.207602 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:20.208156 (+ 554us) server_negotiation.cc:299] Negotiation successful
0808 22:22:20.208844 (+ 688us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":79,"threads_started":1}
I20260808 22:22:20.210609 21041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:20.193956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.253:37039 (local address 127.9.176.195:48487)
0808 22:22:20.195979 (+ 2023us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:20.195991 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:20.196149 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:20.196487 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:20.196489 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:20.196660 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:20.196762 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:20.196769 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:20.197859 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:20.197863 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:20.200081 (+ 2218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:20.200089 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:20.200706 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:20.200709 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:20.200795 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:20.201626 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:20.201646 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:20.203508 (+ 1862us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:22:20.205424 (+ 1916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:20.205430 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:20.205445 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:20.206031 (+ 586us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:20.206569 (+ 538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:20.206572 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:20.206574 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:20.206749 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:20.207230 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:22:20.207236 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:22:20.207524 (+ 288us) client_negotiation.cc:769] Sending connection context
0808 22:22:20.208768 (+ 1244us) client_negotiation.cc:240] Negotiation successful
0808 22:22:20.209372 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1965,"mutex_wait_us":1436,"thread_start_us":73,"threads_started":1}
I20260808 22:22:20.212121 21034 heartbeater.cc:347] Connected to a master server at 127.9.176.253:37039
I20260808 22:22:20.212383 21034 heartbeater.cc:464] Registering TS with master...
I20260808 22:22:20.212957 21034 heartbeater.cc:511] Master 127.9.176.253:37039 requested a full tablet report, sending...
Aug 08 22:22:20 dist-test-slave-ncp9 krb5kdc[19010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227740, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.176.195@KRBTEST.COM for oryx/127.9.176.252@KRBTEST.COM
I20260808 22:22:20.214035 20035 ts_manager.cc:194] Registered new tserver with Master: cf24739aa12a410fa19526e69e0c2d06 (127.9.176.195:37359)
I20260808 22:22:20.214643 21040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:20.192940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:46579 (local address 127.9.176.195:54091)
0808 22:22:20.193442 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:20.193494 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:20.194772 (+ 1278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:20.195516 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:20.195526 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:20.196168 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:20.196662 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:20.196681 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:20.197859 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:20.197863 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:20.199929 (+ 2066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:20.199940 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:20.200787 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:20.200791 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:20.200886 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:20.202752 (+ 1866us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:20.202767 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:20.205680 (+ 2913us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:22:20.212258 (+ 6578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:20.212261 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:20.212262 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:20.212542 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:20.213599 (+ 1057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:20.213602 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:20.213604 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:20.213694 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:20.214161 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:22:20.214163 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:22:20.214251 (+ 88us) client_negotiation.cc:769] Sending connection context
0808 22:22:20.214336 (+ 85us) client_negotiation.cc:240] Negotiation successful
0808 22:22:20.214461 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"mutex_wait_us":1042,"thread_start_us":126,"threads_started":1}
I20260808 22:22:20.215250 19143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:20.192368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.195:54091 (local address 127.9.176.254:46579)
0808 22:22:20.192514 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:20.192519 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:20.193613 (+ 1094us) server_negotiation.cc:407] Connection header received
0808 22:22:20.195016 (+ 1403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:20.195019 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:20.195075 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:20.195135 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:20.196832 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:20.197675 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:20.200203 (+ 2528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:20.200519 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:20.200592 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:20.207192 (+ 6600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:20.207195 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:20.207198 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:22:20.207222 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:20.208751 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:20.212747 (+ 3996us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:20.212750 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:20.212752 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:20.212831 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:20.213800 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:20.213803 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:20.213805 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:20.213993 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:20.214076 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:20.215006 (+ 930us) server_negotiation.cc:299] Negotiation successful
0808 22:22:20.215119 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260808 22:22:20.212214 19077 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34150:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260808 22:22:20.215737 21032 heartbeater.cc:347] Connected to a master server at 127.9.176.254:46579
I20260808 22:22:20.215817 21032 heartbeater.cc:464] Registering TS with master...
I20260808 22:22:20.215966 21032 heartbeater.cc:511] Master 127.9.176.254:46579 requested a full tablet report, sending...
I20260808 22:22:20.216535 19078 ts_manager.cc:194] Re-registered known tserver with Master: cf24739aa12a410fa19526e69e0c2d06 (127.9.176.195:37359)
I20260808 22:22:20.217240 19078 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.176.195@KRBTEST.COM'} at 127.9.176.195:54091
I20260808 22:22:20.218138 20618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:20.192368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.195:36653 (local address 127.9.176.252:34903)
0808 22:22:20.192494 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:20.192499 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:20.194276 (+ 1777us) server_negotiation.cc:407] Connection header received
0808 22:22:20.195014 (+ 738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:20.195018 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:20.195076 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:20.195129 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:20.196833 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:20.197641 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:20.199426 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:20.199719 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:20.199793 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:20.214508 (+ 14715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:20.214512 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:20.214514 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:22:20.214538 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:20.216192 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:20.216699 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:20.216702 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:20.216704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:20.216788 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:20.217236 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:20.217239 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:20.217241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:20.217443 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:20.217535 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:20.217884 (+ 349us) server_negotiation.cc:299] Negotiation successful
0808 22:22:20.218000 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 22:22:20.218564 21047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:20.192946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.252:34903 (local address 127.9.176.195:36653)
0808 22:22:20.194194 (+ 1248us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:20.194206 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:20.194779 (+ 573us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:20.195518 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:20.195526 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:20.195958 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:20.196665 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:20.196683 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:20.198042 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:20.198045 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:20.199311 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:20.199319 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:20.199971 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:20.199975 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:20.200187 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:20.211471 (+ 11284us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:20.211485 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:20.214349 (+ 2864us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:22:20.216312 (+ 1963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:20.216316 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:20.216318 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:20.216610 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:20.216916 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:20.216920 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:20.216922 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:20.217106 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:20.217606 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:22:20.217609 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:22:20.217695 (+ 86us) client_negotiation.cc:769] Sending connection context
0808 22:22:20.218260 (+ 565us) client_negotiation.cc:240] Negotiation successful
0808 22:22:20.218392 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1168,"mutex_wait_us":105,"spinlock_wait_cycles":12160,"thread_start_us":136,"threads_started":1}
I20260808 22:22:20.219236 20235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:20.195560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36088 (local address 127.9.176.253:37039)
0808 22:22:20.208248 (+ 12688us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:20.208251 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:20.208264 (+ 13us) server_negotiation.cc:407] Connection header received
0808 22:22:20.208298 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:20.208301 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:20.208339 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:20.208420 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:20.209183 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:20.209919 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:20.211123 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:20.211423 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:20.211497 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:20.215823 (+ 4326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:20.215827 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:20.215829 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:22:20.215854 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:20.217437 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:20.217880 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:20.217883 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:20.217885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:20.217963 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:20.218308 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:20.218312 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:20.218314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:20.218493 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:20.218604 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:20.218987 (+ 383us) server_negotiation.cc:299] Negotiation successful
0808 22:22:20.219089 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12628,"thread_start_us":101,"threads_started":1}
I20260808 22:22:20.219439 21031 heartbeater.cc:347] Connected to a master server at 127.9.176.252:34903
I20260808 22:22:20.219520 21031 heartbeater.cc:464] Registering TS with master...
I20260808 22:22:20.219699 21031 heartbeater.cc:511] Master 127.9.176.252:34903 requested a full tablet report, sending...
W20260808 22:22:20.219949 19077 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 22:22:20.220321 20329 ts_manager.cc:194] Registered new tserver with Master: cf24739aa12a410fa19526e69e0c2d06 (127.9.176.195:37359)
I20260808 22:22:20.222456 20267 sys_catalog.cc:455] T 00000000000000000000000000000000 P be929da42de644648042b1c514d26d16 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "be929da42de644648042b1c514d26d16" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "be929da42de644648042b1c514d26d16" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 46579 } } peers { permanent_uuid: "28f68f09769f4576ab963f9249cdf9e1" member_type: VOTER last_known_addr { host: "127.9.176.253" port: 37039 } attrs { promote: false } } peers { permanent_uuid: "bacae03153b74596ba962b8db7fedc8f" member_type: NON_VOTER last_known_addr { host: "127.9.176.252" port: 34903 } attrs { promote: true } } }
I20260808 22:22:20.222584 20267 sys_catalog.cc:458] T 00000000000000000000000000000000 P be929da42de644648042b1c514d26d16 [sys.catalog]: This master's current role is: LEADER
I20260808 22:22:20.226083 21063 raft_consensus.cc:1050] T 00000000000000000000000000000000 P be929da42de644648042b1c514d26d16: attempting to promote NON_VOTER bacae03153b74596ba962b8db7fedc8f to VOTER
I20260808 22:22:20.226387 21063 consensus_queue.cc:237] T 00000000000000000000000000000000 P be929da42de644648042b1c514d26d16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "be929da42de644648042b1c514d26d16" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 46579 } } peers { permanent_uuid: "28f68f09769f4576ab963f9249cdf9e1" member_type: VOTER last_known_addr { host: "127.9.176.253" port: 37039 } attrs { promote: false } } peers { permanent_uuid: "bacae03153b74596ba962b8db7fedc8f" member_type: VOTER last_known_addr { host: "127.9.176.252" port: 34903 } attrs { promote: false } }
I20260808 22:22:20.227558 20345 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bacae03153b74596ba962b8db7fedc8f [term 1 LEARNER]: Refusing update from remote peer be929da42de644648042b1c514d26d16: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260808 22:22:20.227909 20053 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 28f68f09769f4576ab963f9249cdf9e1 [term 1 FOLLOWER]: Refusing update from remote peer be929da42de644648042b1c514d26d16: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260808 22:22:20.227939 21065 consensus_queue.cc:1035] T 00000000000000000000000000000000 P be929da42de644648042b1c514d26d16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bacae03153b74596ba962b8db7fedc8f" member_type: VOTER last_known_addr { host: "127.9.176.252" port: 34903 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260808 22:22:20.228397 21065 consensus_queue.cc:1037] T 00000000000000000000000000000000 P be929da42de644648042b1c514d26d16 [LEADER]: Got LMP mismatch error from peer: Peer: permanent_uuid: "28f68f09769f4576ab963f9249cdf9e1" member_type: VOTER last_known_addr { host: "127.9.176.253" port: 37039 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 1.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260808 22:22:20.229804 21059 raft_consensus.cc:2932] T 00000000000000000000000000000000 P be929da42de644648042b1c514d26d16 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, bacae03153b74596ba962b8db7fedc8f (127.9.176.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "be929da42de644648042b1c514d26d16" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 46579 } } peers { permanent_uuid: "28f68f09769f4576ab963f9249cdf9e1" member_type: VOTER last_known_addr { host: "127.9.176.253" port: 37039 } attrs { promote: false } } peers { permanent_uuid: "bacae03153b74596ba962b8db7fedc8f" member_type: VOTER last_known_addr { host: "127.9.176.252" port: 34903 } attrs { promote: false } } }
I20260808 22:22:20.231096 21065 sys_catalog.cc:455] T 00000000000000000000000000000000 P be929da42de644648042b1c514d26d16 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "be929da42de644648042b1c514d26d16" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "be929da42de644648042b1c514d26d16" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 46579 } } peers { permanent_uuid: "28f68f09769f4576ab963f9249cdf9e1" member_type: VOTER last_known_addr { host: "127.9.176.253" port: 37039 } attrs { promote: false } } peers { permanent_uuid: "bacae03153b74596ba962b8db7fedc8f" member_type: VOTER last_known_addr { host: "127.9.176.252" port: 34903 } attrs { promote: false } } }
I20260808 22:22:20.231168 20267 sys_catalog.cc:455] T 00000000000000000000000000000000 P be929da42de644648042b1c514d26d16 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "be929da42de644648042b1c514d26d16" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "be929da42de644648042b1c514d26d16" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 46579 } } peers { permanent_uuid: "28f68f09769f4576ab963f9249cdf9e1" member_type: VOTER last_known_addr { host: "127.9.176.253" port: 37039 } attrs { promote: false } } peers { permanent_uuid: "bacae03153b74596ba962b8db7fedc8f" member_type: VOTER last_known_addr { host: "127.9.176.252" port: 34903 } attrs { promote: false } } }
I20260808 22:22:20.231222 21065 sys_catalog.cc:458] T 00000000000000000000000000000000 P be929da42de644648042b1c514d26d16 [sys.catalog]: This master's current role is: LEADER
I20260808 22:22:20.231237 20267 sys_catalog.cc:458] T 00000000000000000000000000000000 P be929da42de644648042b1c514d26d16 [sys.catalog]: This master's current role is: LEADER
I20260808 22:22:20.231108 21059 sys_catalog.cc:455] T 00000000000000000000000000000000 P be929da42de644648042b1c514d26d16 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "be929da42de644648042b1c514d26d16" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "be929da42de644648042b1c514d26d16" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 46579 } } peers { permanent_uuid: "28f68f09769f4576ab963f9249cdf9e1" member_type: VOTER last_known_addr { host: "127.9.176.253" port: 37039 } attrs { promote: false } } peers { permanent_uuid: "bacae03153b74596ba962b8db7fedc8f" member_type: VOTER last_known_addr { host: "127.9.176.252" port: 34903 } attrs { promote: false } } }
I20260808 22:22:20.231666 21059 sys_catalog.cc:458] T 00000000000000000000000000000000 P be929da42de644648042b1c514d26d16 [sys.catalog]: This master's current role is: LEADER
I20260808 22:22:20.231761 20053 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 28f68f09769f4576ab963f9249cdf9e1 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, bacae03153b74596ba962b8db7fedc8f (127.9.176.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "be929da42de644648042b1c514d26d16" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 46579 } } peers { permanent_uuid: "28f68f09769f4576ab963f9249cdf9e1" member_type: VOTER last_known_addr { host: "127.9.176.253" port: 37039 } attrs { promote: false } } peers { permanent_uuid: "bacae03153b74596ba962b8db7fedc8f" member_type: VOTER last_known_addr { host: "127.9.176.252" port: 34903 } attrs { promote: false } } }
I20260808 22:22:20.231709 20345 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bacae03153b74596ba962b8db7fedc8f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, bacae03153b74596ba962b8db7fedc8f (127.9.176.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "be929da42de644648042b1c514d26d16" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 46579 } } peers { permanent_uuid: "28f68f09769f4576ab963f9249cdf9e1" member_type: VOTER last_known_addr { host: "127.9.176.253" port: 37039 } attrs { promote: false } } peers { permanent_uuid: "bacae03153b74596ba962b8db7fedc8f" member_type: VOTER last_known_addr { host: "127.9.176.252" port: 34903 } attrs { promote: false } } }
I20260808 22:22:20.233547 21080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28f68f09769f4576ab963f9249cdf9e1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "be929da42de644648042b1c514d26d16" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "be929da42de644648042b1c514d26d16" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 46579 } } peers { permanent_uuid: "28f68f09769f4576ab963f9249cdf9e1" member_type: VOTER last_known_addr { host: "127.9.176.253" port: 37039 } attrs { promote: false } } peers { permanent_uuid: "bacae03153b74596ba962b8db7fedc8f" member_type: VOTER last_known_addr { host: "127.9.176.252" port: 34903 } attrs { promote: false } } }
I20260808 22:22:20.233678 21080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28f68f09769f4576ab963f9249cdf9e1 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 22:22:20.233676 20386 sys_catalog.cc:455] T 00000000000000000000000000000000 P bacae03153b74596ba962b8db7fedc8f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "be929da42de644648042b1c514d26d16" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "be929da42de644648042b1c514d26d16" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 46579 } } peers { permanent_uuid: "28f68f09769f4576ab963f9249cdf9e1" member_type: VOTER last_known_addr { host: "127.9.176.253" port: 37039 } attrs { promote: false } } peers { permanent_uuid: "bacae03153b74596ba962b8db7fedc8f" member_type: VOTER last_known_addr { host: "127.9.176.252" port: 34903 } attrs { promote: false } } }
I20260808 22:22:20.233773 20386 sys_catalog.cc:458] T 00000000000000000000000000000000 P bacae03153b74596ba962b8db7fedc8f [sys.catalog]: This master's current role is: FOLLOWER
I20260808 22:22:20.241540 19972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:20.235905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.194:46339 (local address 127.0.0.1:52338)
0808 22:22:20.236322 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:20.236336 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:20.236406 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:20.237717 (+ 1311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:20.237719 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:20.237736 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:22:20.237837 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:20.237843 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:20.238976 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:20.238979 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:20.239852 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:20.239862 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:20.241113 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:20.241116 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:20.241258 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:20.241282 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 22:22:20.241351 (+ 69us) client_negotiation.cc:240] Negotiation successful
0808 22:22:20.241419 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"mutex_wait_us":25}
I20260808 22:22:20.241745 21082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:20.236102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52338 (local address 127.9.176.194:46339)
0808 22:22:20.237037 (+ 935us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:20.237047 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:20.237069 (+ 22us) server_negotiation.cc:407] Connection header received
0808 22:22:20.237148 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:20.237164 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:20.237381 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:20.237756 (+ 375us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:22:20.237937 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:20.238856 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:20.240118 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:20.241005 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:20.241103 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:20.241253 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:20.241383 (+ 130us) server_negotiation.cc:299] Negotiation successful
0808 22:22:20.241472 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":833,"thread_start_us":107,"threads_started":1}
I20260808 22:22:20.242355 21081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:20.236084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.193:34919 (local address 127.0.0.1:42816)
0808 22:22:20.237500 (+ 1416us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:20.237514 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:20.237594 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:20.238069 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:20.238072 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:20.238091 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:22:20.238174 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:20.238179 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:20.239532 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:20.239536 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:20.240610 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:20.240621 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:20.241830 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:20.241834 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:20.242034 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:20.242070 (+ 36us) client_negotiation.cc:769] Sending connection context
0808 22:22:20.242145 (+ 75us) client_negotiation.cc:240] Negotiation successful
0808 22:22:20.242231 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":109,"threads_started":1}
I20260808 22:22:20.242504 21084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:20.236590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42816 (local address 127.9.176.193:34919)
0808 22:22:20.237718 (+ 1128us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:20.237732 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:20.237752 (+ 20us) server_negotiation.cc:407] Connection header received
0808 22:22:20.237813 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:20.237816 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:20.237953 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:20.238074 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:22:20.238300 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:20.239412 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:20.240722 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:20.241543 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:20.241639 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:20.241767 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:20.242199 (+ 432us) server_negotiation.cc:299] Negotiation successful
0808 22:22:20.242293 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1043,"thread_start_us":92,"threads_started":1}
I20260808 22:22:20.243005 21083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:20.236360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.195:37359 (local address 127.0.0.1:59904)
0808 22:22:20.236566 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:20.236580 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:20.236681 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:20.238721 (+ 2040us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:20.238724 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:20.238741 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:22:20.238827 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:20.238832 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:20.239914 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:20.239918 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:20.241228 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:20.241239 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:20.242536 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:20.242540 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:20.242701 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:20.242727 (+ 26us) client_negotiation.cc:769] Sending connection context
0808 22:22:20.242794 (+ 67us) client_negotiation.cc:240] Negotiation successful
0808 22:22:20.242883 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":21,"thread_start_us":94,"threads_started":1}
I20260808 22:22:20.243464 21086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:20.236741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59904 (local address 127.9.176.195:37359)
0808 22:22:20.238316 (+ 1575us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:20.238327 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:20.238349 (+ 22us) server_negotiation.cc:407] Connection header received
0808 22:22:20.238413 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:20.238418 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:20.238591 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:20.238733 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:22:20.238920 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:20.239813 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:20.241367 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:20.242252 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:20.242343 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:20.242473 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:20.243254 (+ 781us) server_negotiation.cc:299] Negotiation successful
0808 22:22:20.243347 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1478,"thread_start_us":94,"threads_started":1}
I20260808 22:22:20.244752 20723 tablet_service.cc:1467] Processing CreateTablet for tablet dbf2949520324757934a7d828c15d942 (DEFAULT_TABLE table=test-table [id=f07028d3515746e6af0a3d7f8aedc71a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:22:20.245265 20503 tablet_service.cc:1467] Processing CreateTablet for tablet dbf2949520324757934a7d828c15d942 (DEFAULT_TABLE table=test-table [id=f07028d3515746e6af0a3d7f8aedc71a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:22:20.245837 20936 tablet_service.cc:1467] Processing CreateTablet for tablet dbf2949520324757934a7d828c15d942 (DEFAULT_TABLE table=test-table [id=f07028d3515746e6af0a3d7f8aedc71a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:22:20.246210 20723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbf2949520324757934a7d828c15d942. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:22:20.246236 20503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbf2949520324757934a7d828c15d942. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:22:20.246714 20936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbf2949520324757934a7d828c15d942. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:22:20.253466 21088 tablet_bootstrap.cc:492] T dbf2949520324757934a7d828c15d942 P cf24739aa12a410fa19526e69e0c2d06: Bootstrap starting.
I20260808 22:22:20.254765 21089 tablet_bootstrap.cc:492] T dbf2949520324757934a7d828c15d942 P 41b49dbbd6a04fce85c78e62b80ba81e: Bootstrap starting.
I20260808 22:22:20.256417 21088 tablet_bootstrap.cc:654] T dbf2949520324757934a7d828c15d942 P cf24739aa12a410fa19526e69e0c2d06: Neither blocks nor log segments found. Creating new log.
I20260808 22:22:20.256417 21089 tablet_bootstrap.cc:654] T dbf2949520324757934a7d828c15d942 P 41b49dbbd6a04fce85c78e62b80ba81e: Neither blocks nor log segments found. Creating new log.
I20260808 22:22:20.257180 21089 log.cc:824] T dbf2949520324757934a7d828c15d942 P 41b49dbbd6a04fce85c78e62b80ba81e: Log is configured to *not* fsync() on all Append() calls
I20260808 22:22:20.257238 21090 tablet_bootstrap.cc:492] T dbf2949520324757934a7d828c15d942 P 39101d245faa4b6eb281ffc89ef69bf6: Bootstrap starting.
I20260808 22:22:20.257551 21088 log.cc:824] T dbf2949520324757934a7d828c15d942 P cf24739aa12a410fa19526e69e0c2d06: Log is configured to *not* fsync() on all Append() calls
I20260808 22:22:20.258982 21090 tablet_bootstrap.cc:654] T dbf2949520324757934a7d828c15d942 P 39101d245faa4b6eb281ffc89ef69bf6: Neither blocks nor log segments found. Creating new log.
I20260808 22:22:20.259654 21090 log.cc:824] T dbf2949520324757934a7d828c15d942 P 39101d245faa4b6eb281ffc89ef69bf6: Log is configured to *not* fsync() on all Append() calls
I20260808 22:22:20.260393 21088 tablet_bootstrap.cc:492] T dbf2949520324757934a7d828c15d942 P cf24739aa12a410fa19526e69e0c2d06: No bootstrap required, opened a new log
I20260808 22:22:20.260393 21089 tablet_bootstrap.cc:492] T dbf2949520324757934a7d828c15d942 P 41b49dbbd6a04fce85c78e62b80ba81e: No bootstrap required, opened a new log
I20260808 22:22:20.260581 21089 ts_tablet_manager.cc:1397] T dbf2949520324757934a7d828c15d942 P 41b49dbbd6a04fce85c78e62b80ba81e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 22:22:20.260581 21088 ts_tablet_manager.cc:1397] T dbf2949520324757934a7d828c15d942 P cf24739aa12a410fa19526e69e0c2d06: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20260808 22:22:20.261777 21090 tablet_bootstrap.cc:492] T dbf2949520324757934a7d828c15d942 P 39101d245faa4b6eb281ffc89ef69bf6: No bootstrap required, opened a new log
I20260808 22:22:20.261969 21090 ts_tablet_manager.cc:1397] T dbf2949520324757934a7d828c15d942 P 39101d245faa4b6eb281ffc89ef69bf6: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260808 22:22:20.265089 21088 raft_consensus.cc:348] T dbf2949520324757934a7d828c15d942 P cf24739aa12a410fa19526e69e0c2d06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b49dbbd6a04fce85c78e62b80ba81e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 46339 } } peers { permanent_uuid: "39101d245faa4b6eb281ffc89ef69bf6" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34919 } } peers { permanent_uuid: "cf24739aa12a410fa19526e69e0c2d06" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37359 } }
I20260808 22:22:20.265089 21089 raft_consensus.cc:348] T dbf2949520324757934a7d828c15d942 P 41b49dbbd6a04fce85c78e62b80ba81e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b49dbbd6a04fce85c78e62b80ba81e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 46339 } } peers { permanent_uuid: "39101d245faa4b6eb281ffc89ef69bf6" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34919 } } peers { permanent_uuid: "cf24739aa12a410fa19526e69e0c2d06" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37359 } }
I20260808 22:22:20.265352 21089 raft_consensus.cc:374] T dbf2949520324757934a7d828c15d942 P 41b49dbbd6a04fce85c78e62b80ba81e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:22:20.265352 21088 raft_consensus.cc:374] T dbf2949520324757934a7d828c15d942 P cf24739aa12a410fa19526e69e0c2d06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:22:20.265410 21089 raft_consensus.cc:729] T dbf2949520324757934a7d828c15d942 P 41b49dbbd6a04fce85c78e62b80ba81e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41b49dbbd6a04fce85c78e62b80ba81e, State: Initialized, Role: FOLLOWER
I20260808 22:22:20.265410 21088 raft_consensus.cc:729] T dbf2949520324757934a7d828c15d942 P cf24739aa12a410fa19526e69e0c2d06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf24739aa12a410fa19526e69e0c2d06, State: Initialized, Role: FOLLOWER
I20260808 22:22:20.265296 21090 raft_consensus.cc:348] T dbf2949520324757934a7d828c15d942 P 39101d245faa4b6eb281ffc89ef69bf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b49dbbd6a04fce85c78e62b80ba81e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 46339 } } peers { permanent_uuid: "39101d245faa4b6eb281ffc89ef69bf6" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34919 } } peers { permanent_uuid: "cf24739aa12a410fa19526e69e0c2d06" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37359 } }
I20260808 22:22:20.265538 21090 raft_consensus.cc:374] T dbf2949520324757934a7d828c15d942 P 39101d245faa4b6eb281ffc89ef69bf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:22:20.265612 21090 raft_consensus.cc:729] T dbf2949520324757934a7d828c15d942 P 39101d245faa4b6eb281ffc89ef69bf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39101d245faa4b6eb281ffc89ef69bf6, State: Initialized, Role: FOLLOWER
I20260808 22:22:20.265865 21089 consensus_queue.cc:260] T dbf2949520324757934a7d828c15d942 P 41b49dbbd6a04fce85c78e62b80ba81e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b49dbbd6a04fce85c78e62b80ba81e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 46339 } } peers { permanent_uuid: "39101d245faa4b6eb281ffc89ef69bf6" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34919 } } peers { permanent_uuid: "cf24739aa12a410fa19526e69e0c2d06" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37359 } }
I20260808 22:22:20.265865 21088 consensus_queue.cc:260] T dbf2949520324757934a7d828c15d942 P cf24739aa12a410fa19526e69e0c2d06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b49dbbd6a04fce85c78e62b80ba81e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 46339 } } peers { permanent_uuid: "39101d245faa4b6eb281ffc89ef69bf6" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34919 } } peers { permanent_uuid: "cf24739aa12a410fa19526e69e0c2d06" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37359 } }
I20260808 22:22:20.265985 21090 consensus_queue.cc:260] T dbf2949520324757934a7d828c15d942 P 39101d245faa4b6eb281ffc89ef69bf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b49dbbd6a04fce85c78e62b80ba81e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 46339 } } peers { permanent_uuid: "39101d245faa4b6eb281ffc89ef69bf6" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34919 } } peers { permanent_uuid: "cf24739aa12a410fa19526e69e0c2d06" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37359 } }
I20260808 22:22:20.266847 21089 ts_tablet_manager.cc:1428] T dbf2949520324757934a7d828c15d942 P 41b49dbbd6a04fce85c78e62b80ba81e: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260808 22:22:20.266847 21088 ts_tablet_manager.cc:1428] T dbf2949520324757934a7d828c15d942 P cf24739aa12a410fa19526e69e0c2d06: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260808 22:22:20.266909 21090 ts_tablet_manager.cc:1428] T dbf2949520324757934a7d828c15d942 P 39101d245faa4b6eb281ffc89ef69bf6: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260808 22:22:20.267222 20616 heartbeater.cc:503] Master 127.9.176.254:46579 was elected leader, sending a full tablet report...
I20260808 22:22:20.267717 21032 heartbeater.cc:503] Master 127.9.176.254:46579 was elected leader, sending a full tablet report...
I20260808 22:22:20.270593 20822 heartbeater.cc:503] Master 127.9.176.254:46579 was elected leader, sending a full tablet report...
W20260808 22:22:20.310869 20622 tablet.cc:2369] T dbf2949520324757934a7d828c15d942 P 39101d245faa4b6eb281ffc89ef69bf6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 22:22:20.332644 21100 raft_consensus.cc:482] T dbf2949520324757934a7d828c15d942 P 39101d245faa4b6eb281ffc89ef69bf6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 22:22:20.332857 21100 raft_consensus.cc:504] T dbf2949520324757934a7d828c15d942 P 39101d245faa4b6eb281ffc89ef69bf6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b49dbbd6a04fce85c78e62b80ba81e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 46339 } } peers { permanent_uuid: "39101d245faa4b6eb281ffc89ef69bf6" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34919 } } peers { permanent_uuid: "cf24739aa12a410fa19526e69e0c2d06" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37359 } }
I20260808 22:22:20.334142 21100 leader_election.cc:288] T dbf2949520324757934a7d828c15d942 P 39101d245faa4b6eb281ffc89ef69bf6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 41b49dbbd6a04fce85c78e62b80ba81e (127.9.176.194:46339), cf24739aa12a410fa19526e69e0c2d06 (127.9.176.195:37359)
I20260808 22:22:20.338313 21086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:20.334495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.193:34485 (local address 127.9.176.195:37359)
0808 22:22:20.334621 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:20.334627 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:20.334771 (+ 144us) server_negotiation.cc:407] Connection header received
0808 22:22:20.334929 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:20.334932 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:20.334981 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:20.335044 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:22:20.335308 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:20.336089 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:20.337380 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:20.337799 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:20.337910 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:20.337967 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:20.338121 (+ 154us) server_negotiation.cc:299] Negotiation successful
0808 22:22:20.338193 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260808 22:22:20.338330 21101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:20.334315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.195:37359 (local address 127.9.176.193:34485)
0808 22:22:20.334674 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:20.334688 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:20.334795 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:20.335061 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:20.335064 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:20.335079 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:22:20.335189 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:20.335196 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:20.336201 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:20.336204 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:20.337286 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:20.337294 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:20.337909 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:20.337913 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:20.338010 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:20.338030 (+ 20us) client_negotiation.cc:769] Sending connection context
0808 22:22:20.338097 (+ 67us) client_negotiation.cc:240] Negotiation successful
0808 22:22:20.338181 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":67,"threads_started":1}
I20260808 22:22:20.338804 21082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:20.334491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.193:45763 (local address 127.9.176.194:46339)
0808 22:22:20.334631 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:20.334636 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:20.334838 (+ 202us) server_negotiation.cc:407] Connection header received
0808 22:22:20.334979 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:20.334982 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:20.335030 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:20.335102 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:22:20.335378 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:20.335871 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:20.337351 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:20.338013 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:20.338130 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:20.338178 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:20.338610 (+ 432us) server_negotiation.cc:299] Negotiation successful
0808 22:22:20.338690 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 22:22:20.338907 20963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbf2949520324757934a7d828c15d942" candidate_uuid: "39101d245faa4b6eb281ffc89ef69bf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf24739aa12a410fa19526e69e0c2d06" is_pre_election: true
I20260808 22:22:20.339159 21102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:20.334315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.194:46339 (local address 127.9.176.193:45763)
0808 22:22:20.334763 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:20.334777 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:20.334878 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:20.335172 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:20.335174 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:20.335189 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:22:20.335273 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:20.335278 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:20.335973 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:20.335976 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:20.337219 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:20.337230 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:20.338253 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:20.338256 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:20.338359 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:20.338501 (+ 142us) client_negotiation.cc:769] Sending connection context
0808 22:22:20.338936 (+ 435us) client_negotiation.cc:240] Negotiation successful
0808 22:22:20.339013 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"mutex_wait_us":219,"thread_start_us":50,"threads_started":1}
I20260808 22:22:20.339221 20963 raft_consensus.cc:2454] T dbf2949520324757934a7d828c15d942 P cf24739aa12a410fa19526e69e0c2d06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39101d245faa4b6eb281ffc89ef69bf6 in term 0.
I20260808 22:22:20.339668 20429 leader_election.cc:302] T dbf2949520324757934a7d828c15d942 P 39101d245faa4b6eb281ffc89ef69bf6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39101d245faa4b6eb281ffc89ef69bf6, cf24739aa12a410fa19526e69e0c2d06; no voters:
I20260808 22:22:20.339715 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbf2949520324757934a7d828c15d942" candidate_uuid: "39101d245faa4b6eb281ffc89ef69bf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41b49dbbd6a04fce85c78e62b80ba81e" is_pre_election: true
I20260808 22:22:20.339956 21100 raft_consensus.cc:2781] T dbf2949520324757934a7d828c15d942 P 39101d245faa4b6eb281ffc89ef69bf6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 22:22:20.339977 20750 raft_consensus.cc:2454] T dbf2949520324757934a7d828c15d942 P 41b49dbbd6a04fce85c78e62b80ba81e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39101d245faa4b6eb281ffc89ef69bf6 in term 0.
I20260808 22:22:20.340059 21100 raft_consensus.cc:482] T dbf2949520324757934a7d828c15d942 P 39101d245faa4b6eb281ffc89ef69bf6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 22:22:20.340102 21100 raft_consensus.cc:3037] T dbf2949520324757934a7d828c15d942 P 39101d245faa4b6eb281ffc89ef69bf6 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:22:20.341269 21100 raft_consensus.cc:504] T dbf2949520324757934a7d828c15d942 P 39101d245faa4b6eb281ffc89ef69bf6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b49dbbd6a04fce85c78e62b80ba81e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 46339 } } peers { permanent_uuid: "39101d245faa4b6eb281ffc89ef69bf6" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34919 } } peers { permanent_uuid: "cf24739aa12a410fa19526e69e0c2d06" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37359 } }
I20260808 22:22:20.341602 21100 leader_election.cc:288] T dbf2949520324757934a7d828c15d942 P 39101d245faa4b6eb281ffc89ef69bf6 [CANDIDATE]: Term 1 election: Requested vote from peers 41b49dbbd6a04fce85c78e62b80ba81e (127.9.176.194:46339), cf24739aa12a410fa19526e69e0c2d06 (127.9.176.195:37359)
I20260808 22:22:20.341920 20963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbf2949520324757934a7d828c15d942" candidate_uuid: "39101d245faa4b6eb281ffc89ef69bf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf24739aa12a410fa19526e69e0c2d06"
I20260808 22:22:20.341938 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbf2949520324757934a7d828c15d942" candidate_uuid: "39101d245faa4b6eb281ffc89ef69bf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41b49dbbd6a04fce85c78e62b80ba81e"
I20260808 22:22:20.342049 20963 raft_consensus.cc:3037] T dbf2949520324757934a7d828c15d942 P cf24739aa12a410fa19526e69e0c2d06 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:22:20.342103 20750 raft_consensus.cc:3037] T dbf2949520324757934a7d828c15d942 P 41b49dbbd6a04fce85c78e62b80ba81e [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:22:20.343130 20963 raft_consensus.cc:2454] T dbf2949520324757934a7d828c15d942 P cf24739aa12a410fa19526e69e0c2d06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39101d245faa4b6eb281ffc89ef69bf6 in term 1.
I20260808 22:22:20.343300 20750 raft_consensus.cc:2454] T dbf2949520324757934a7d828c15d942 P 41b49dbbd6a04fce85c78e62b80ba81e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39101d245faa4b6eb281ffc89ef69bf6 in term 1.
I20260808 22:22:20.343391 20429 leader_election.cc:302] T dbf2949520324757934a7d828c15d942 P 39101d245faa4b6eb281ffc89ef69bf6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39101d245faa4b6eb281ffc89ef69bf6, cf24739aa12a410fa19526e69e0c2d06; no voters:
I20260808 22:22:20.343550 21100 raft_consensus.cc:2781] T dbf2949520324757934a7d828c15d942 P 39101d245faa4b6eb281ffc89ef69bf6 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:22:20.343750 21100 raft_consensus.cc:686] T dbf2949520324757934a7d828c15d942 P 39101d245faa4b6eb281ffc89ef69bf6 [term 1 LEADER]: Becoming Leader. State: Replica: 39101d245faa4b6eb281ffc89ef69bf6, State: Running, Role: LEADER
I20260808 22:22:20.344079 21100 consensus_queue.cc:237] T dbf2949520324757934a7d828c15d942 P 39101d245faa4b6eb281ffc89ef69bf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b49dbbd6a04fce85c78e62b80ba81e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 46339 } } peers { permanent_uuid: "39101d245faa4b6eb281ffc89ef69bf6" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34919 } } peers { permanent_uuid: "cf24739aa12a410fa19526e69e0c2d06" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37359 } }
I20260808 22:22:20.347227 19078 catalog_manager.cc:5495] T dbf2949520324757934a7d828c15d942 P 39101d245faa4b6eb281ffc89ef69bf6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39101d245faa4b6eb281ffc89ef69bf6 (127.9.176.193). New cstate: current_term: 1 leader_uuid: "39101d245faa4b6eb281ffc89ef69bf6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b49dbbd6a04fce85c78e62b80ba81e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 46339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39101d245faa4b6eb281ffc89ef69bf6" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf24739aa12a410fa19526e69e0c2d06" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37359 } health_report { overall_health: UNKNOWN } } }
I20260808 22:22:20.359524 21084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:20.356171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42818 (local address 127.9.176.193:34919)
0808 22:22:20.356275 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:20.356280 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:20.356312 (+ 32us) server_negotiation.cc:407] Connection header received
0808 22:22:20.356481 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:20.356483 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:20.356534 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:20.356618 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:22:20.356872 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:20.357388 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:20.358199 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:20.358417 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:20.358502 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:20.358789 (+ 287us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:22:20.359176 (+ 387us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:22:20.359282 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:20.359351 (+ 69us) server_negotiation.cc:299] Negotiation successful
0808 22:22:20.359405 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260808 22:22:20.364964 20963 raft_consensus.cc:1261] T dbf2949520324757934a7d828c15d942 P cf24739aa12a410fa19526e69e0c2d06 [term 1 FOLLOWER]: Refusing update from remote peer 39101d245faa4b6eb281ffc89ef69bf6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:22:20.364971 20750 raft_consensus.cc:1261] T dbf2949520324757934a7d828c15d942 P 41b49dbbd6a04fce85c78e62b80ba81e [term 1 FOLLOWER]: Refusing update from remote peer 39101d245faa4b6eb281ffc89ef69bf6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:22:20.365590 21100 consensus_queue.cc:1035] T dbf2949520324757934a7d828c15d942 P 39101d245faa4b6eb281ffc89ef69bf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf24739aa12a410fa19526e69e0c2d06" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:22:20.365784 21103 consensus_queue.cc:1035] T dbf2949520324757934a7d828c15d942 P 39101d245faa4b6eb281ffc89ef69bf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41b49dbbd6a04fce85c78e62b80ba81e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 46339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:22:20.375334 21111 mvcc.cc:205] Tried to move back new op lower bound from 7316388824528449536 to 7316388824451764224. Current Snapshot: MvccSnapshot[applied={T|T < 7316388824528449536}]
I20260808 22:22:20.379503 21112 mvcc.cc:205] Tried to move back new op lower bound from 7316388824528449536 to 7316388824451764224. Current Snapshot: MvccSnapshot[applied={T|T < 7316388824528449536}]
I20260808 22:22:20.384263 19078 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34150:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:22:20.384390 19078 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34150:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:22:20.388019 19078 catalog_manager.cc:5782] T 00000000000000000000000000000000 P be929da42de644648042b1c514d26d16: Sending DeleteTablet for 3 replicas of tablet dbf2949520324757934a7d828c15d942
I20260808 22:22:20.389092 20503 tablet_service.cc:1514] Processing DeleteTablet for tablet dbf2949520324757934a7d828c15d942 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:22:20 UTC) from {username='oryx', principal='oryx/127.9.176.254@KRBTEST.COM'} at 127.0.0.1:42816
I20260808 22:22:20.389088 20723 tablet_service.cc:1514] Processing DeleteTablet for tablet dbf2949520324757934a7d828c15d942 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:22:20 UTC) from {username='oryx', principal='oryx/127.9.176.254@KRBTEST.COM'} at 127.0.0.1:52338
I20260808 22:22:20.389637 20936 tablet_service.cc:1514] Processing DeleteTablet for tablet dbf2949520324757934a7d828c15d942 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:22:20 UTC) from {username='oryx', principal='oryx/127.9.176.254@KRBTEST.COM'} at 127.0.0.1:59904
I20260808 22:22:20.389954 21120 tablet_replica.cc:330] T dbf2949520324757934a7d828c15d942 P 39101d245faa4b6eb281ffc89ef69bf6: stopping tablet replica
I20260808 22:22:20.390242 21122 tablet_replica.cc:330] T dbf2949520324757934a7d828c15d942 P cf24739aa12a410fa19526e69e0c2d06: stopping tablet replica
I20260808 22:22:20.390254 21120 raft_consensus.cc:2229] T dbf2949520324757934a7d828c15d942 P 39101d245faa4b6eb281ffc89ef69bf6 [term 1 LEADER]: Raft consensus shutting down.
I20260808 22:22:20.390583 21122 raft_consensus.cc:2229] T dbf2949520324757934a7d828c15d942 P cf24739aa12a410fa19526e69e0c2d06 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:22:20.390587 21120 raft_consensus.cc:2258] T dbf2949520324757934a7d828c15d942 P 39101d245faa4b6eb281ffc89ef69bf6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:22:20.390950 21122 raft_consensus.cc:2258] T dbf2949520324757934a7d828c15d942 P cf24739aa12a410fa19526e69e0c2d06 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:22:20.391848 21121 tablet_replica.cc:330] T dbf2949520324757934a7d828c15d942 P 41b49dbbd6a04fce85c78e62b80ba81e: stopping tablet replica
I20260808 22:22:20.392067 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 19030
I20260808 22:22:20.398056 21120 ts_tablet_manager.cc:1905] T dbf2949520324757934a7d828c15d942 P 39101d245faa4b6eb281ffc89ef69bf6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:22:20.392127 21121 raft_consensus.cc:2229] T dbf2949520324757934a7d828c15d942 P 41b49dbbd6a04fce85c78e62b80ba81e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:22:20.399304 21121 raft_consensus.cc:2258] T dbf2949520324757934a7d828c15d942 P 41b49dbbd6a04fce85c78e62b80ba81e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:22:20.400846 21121 ts_tablet_manager.cc:1905] T dbf2949520324757934a7d828c15d942 P 41b49dbbd6a04fce85c78e62b80ba81e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:22:20.401225 21120 ts_tablet_manager.cc:1918] T dbf2949520324757934a7d828c15d942 P 39101d245faa4b6eb281ffc89ef69bf6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 22:22:20.401369 21120 log.cc:1196] T dbf2949520324757934a7d828c15d942 P 39101d245faa4b6eb281ffc89ef69bf6: Deleting WAL directory at /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-0/wal/wals/dbf2949520324757934a7d828c15d942
I20260808 22:22:20.401758 21120 ts_tablet_manager.cc:1939] T dbf2949520324757934a7d828c15d942 P 39101d245faa4b6eb281ffc89ef69bf6: Deleting consensus metadata
I20260808 22:22:20.403342 21121 ts_tablet_manager.cc:1918] T dbf2949520324757934a7d828c15d942 P 41b49dbbd6a04fce85c78e62b80ba81e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 22:22:20.403450 21121 log.cc:1196] T dbf2949520324757934a7d828c15d942 P 41b49dbbd6a04fce85c78e62b80ba81e: Deleting WAL directory at /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-1/wal/wals/dbf2949520324757934a7d828c15d942
I20260808 22:22:20.403764 21121 ts_tablet_manager.cc:1939] T dbf2949520324757934a7d828c15d942 P 41b49dbbd6a04fce85c78e62b80ba81e: Deleting consensus metadata
I20260808 22:22:20.404114 21122 ts_tablet_manager.cc:1905] T dbf2949520324757934a7d828c15d942 P cf24739aa12a410fa19526e69e0c2d06: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:22:20.406653 21122 ts_tablet_manager.cc:1918] T dbf2949520324757934a7d828c15d942 P cf24739aa12a410fa19526e69e0c2d06: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 22:22:20.406880 21122 log.cc:1196] T dbf2949520324757934a7d828c15d942 P cf24739aa12a410fa19526e69e0c2d06: Deleting WAL directory at /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-2/wal/wals/dbf2949520324757934a7d828c15d942
W20260808 22:22:20.407179 20645 connection.cc:535] server connection from 127.0.0.1:52338 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260808 22:22:20.407327 20426 connection.cc:535] server connection from 127.0.0.1:42816 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260808 22:22:20.407860 21122 ts_tablet_manager.cc:1939] T dbf2949520324757934a7d828c15d942 P cf24739aa12a410fa19526e69e0c2d06: Deleting consensus metadata
W20260808 22:22:20.409194 20864 connection.cc:441] server connection from 127.0.0.1:59904 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59904 (request call id 1) could send its response
I20260808 22:22:20.412787 20235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:20.391136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36102 (local address 127.9.176.253:37039)
0808 22:22:20.391241 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:20.391245 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:20.391523 (+ 278us) server_negotiation.cc:407] Connection header received
0808 22:22:20.397695 (+ 6172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:20.397699 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:20.397755 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:20.397809 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:20.404838 (+ 7029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:20.405674 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:20.406792 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:20.407107 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:20.407185 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:20.409284 (+ 2099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:20.409288 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:20.409291 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:22:20.409316 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:20.410923 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:20.411384 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:20.411387 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:20.411389 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:20.411469 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:20.411799 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:20.411803 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:20.411805 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:20.412039 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:20.412177 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:20.412506 (+ 329us) server_negotiation.cc:299] Negotiation successful
0808 22:22:20.412629 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260808 22:22:20.419724 21050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:20.410976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37126 (local address 127.9.176.252:34903)
0808 22:22:20.411121 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:20.411125 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:20.411766 (+ 641us) server_negotiation.cc:407] Connection header received
0808 22:22:20.411927 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:20.411930 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:20.411978 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:20.412041 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:20.413060 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:20.413758 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:20.414715 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:20.414952 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:20.415050 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:20.416262 (+ 1212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:20.416266 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:20.416269 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:22:20.416294 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:20.417741 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:20.418330 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:20.418334 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:20.418336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:20.418425 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:20.418729 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:20.418732 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:20.418734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:20.418942 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:20.419071 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:20.419399 (+ 328us) server_negotiation.cc:299] Negotiation successful
0808 22:22:20.419557 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260808 22:22:20.421736 20618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:20.390779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37116 (local address 127.9.176.252:34903)
0808 22:22:20.390893 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:20.390898 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:20.413358 (+ 22460us) server_negotiation.cc:407] Connection header received
0808 22:22:20.413537 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:20.413541 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:20.413587 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:20.413671 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:20.414395 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:20.415170 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:20.416566 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:20.416925 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:20.416997 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:20.418513 (+ 1516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:20.418517 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:20.418519 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:22:20.418569 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:20.420097 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:20.420548 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:20.420551 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:20.420552 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:20.420632 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:20.420965 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:20.420970 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:20.420972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:20.421188 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:20.421313 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:20.421487 (+ 174us) server_negotiation.cc:299] Negotiation successful
0808 22:22:20.421594 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260808 22:22:20.431551 20235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:20.425474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36118 (local address 127.9.176.253:37039)
0808 22:22:20.425576 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:20.425580 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:20.425592 (+ 12us) server_negotiation.cc:407] Connection header received
0808 22:22:20.425630 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:20.425633 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:20.425670 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:20.425731 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:20.426297 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:20.426774 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:20.427487 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:20.427695 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:20.427778 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:20.428917 (+ 1139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:20.428922 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:20.428925 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:22:20.428949 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:20.430136 (+ 1187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:20.430508 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:20.430511 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:20.430513 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:20.430584 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:20.430824 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:20.430828 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:20.430829 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:20.430991 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:20.431099 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:20.431316 (+ 217us) server_negotiation.cc:299] Negotiation successful
0808 22:22:20.431413 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 22:22:21.378324 21137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:21.377854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:46579 (local address 127.9.176.193:46737)
0808 22:22:21.378098 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:21.378221 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":92,"threads_started":1}
W20260808 22:22:21.378652 20616 heartbeater.cc:644] Failed to heartbeat to 127.9.176.254:46579 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.9.176.254:46579: connect: Connection refused (error 111)
I20260808 22:22:21.380203 21138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:21.379630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:46579 (local address 127.9.176.194:35487)
0808 22:22:21.379983 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:21.380081 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":69,"threads_started":1}
W20260808 22:22:21.381089 20822 heartbeater.cc:644] Failed to heartbeat to 127.9.176.254:46579 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.9.176.254:46579: connect: Connection refused (error 111)
I20260808 22:22:21.381381 21139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:21.380691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:46579 (local address 127.9.176.195:34753)
0808 22:22:21.381202 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:21.381293 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":73,"threads_started":1}
W20260808 22:22:21.381609 21032 heartbeater.cc:644] Failed to heartbeat to 127.9.176.254:46579 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.9.176.254:46579: connect: Connection refused (error 111)
I20260808 22:22:21.889014 21140 raft_consensus.cc:482] T 00000000000000000000000000000000 P 28f68f09769f4576ab963f9249cdf9e1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader be929da42de644648042b1c514d26d16)
I20260808 22:22:21.889155 21140 raft_consensus.cc:504] T 00000000000000000000000000000000 P 28f68f09769f4576ab963f9249cdf9e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "be929da42de644648042b1c514d26d16" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 46579 } } peers { permanent_uuid: "28f68f09769f4576ab963f9249cdf9e1" member_type: VOTER last_known_addr { host: "127.9.176.253" port: 37039 } attrs { promote: false } } peers { permanent_uuid: "bacae03153b74596ba962b8db7fedc8f" member_type: VOTER last_known_addr { host: "127.9.176.252" port: 34903 } attrs { promote: false } }
I20260808 22:22:21.889166 21142 raft_consensus.cc:482] T 00000000000000000000000000000000 P bacae03153b74596ba962b8db7fedc8f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader be929da42de644648042b1c514d26d16)
I20260808 22:22:21.889364 21142 raft_consensus.cc:504] T 00000000000000000000000000000000 P bacae03153b74596ba962b8db7fedc8f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "be929da42de644648042b1c514d26d16" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 46579 } } peers { permanent_uuid: "28f68f09769f4576ab963f9249cdf9e1" member_type: VOTER last_known_addr { host: "127.9.176.253" port: 37039 } attrs { promote: false } } peers { permanent_uuid: "bacae03153b74596ba962b8db7fedc8f" member_type: VOTER last_known_addr { host: "127.9.176.252" port: 34903 } attrs { promote: false } }
I20260808 22:22:21.889642 21140 leader_election.cc:288] T 00000000000000000000000000000000 P 28f68f09769f4576ab963f9249cdf9e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers be929da42de644648042b1c514d26d16 (127.9.176.254:46579), bacae03153b74596ba962b8db7fedc8f (127.9.176.252:34903)
I20260808 22:22:21.890269 21144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:21.889754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:46579 (local address 127.0.0.1:34582)
0808 22:22:21.890085 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:21.890173 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":57,"threads_started":1}
I20260808 22:22:21.890475 21142 leader_election.cc:288] T 00000000000000000000000000000000 P bacae03153b74596ba962b8db7fedc8f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers be929da42de644648042b1c514d26d16 (127.9.176.254:46579), 28f68f09769f4576ab963f9249cdf9e1 (127.9.176.253:37039)
W20260808 22:22:21.890501 20013 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.176.254:46579: connect: Connection refused (error 111)
I20260808 22:22:21.890930 21149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:21.890553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:46579 (local address 127.0.0.1:34592)
0808 22:22:21.890767 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:21.890843 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":87,"threads_started":1}
W20260808 22:22:21.891108 20304 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.176.254:46579: connect: Connection refused (error 111)
I20260808 22:22:21.891414 21144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:21.891030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:46579 (local address 127.0.0.1:34606)
0808 22:22:21.891297 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:21.891350 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208}
W20260808 22:22:21.891654 20013 leader_election.cc:334] T 00000000000000000000000000000000 P 28f68f09769f4576ab963f9249cdf9e1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer be929da42de644648042b1c514d26d16 (127.9.176.254:46579): Network error: Client connection negotiation failed: client connection to 127.9.176.254:46579: connect: Connection refused (error 111)
I20260808 22:22:21.892025 21152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:21.891805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:46579 (local address 127.0.0.1:34610)
0808 22:22:21.891908 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:21.891962 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260808 22:22:21.892236 20304 leader_election.cc:334] T 00000000000000000000000000000000 P bacae03153b74596ba962b8db7fedc8f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer be929da42de644648042b1c514d26d16 (127.9.176.254:46579): Network error: Client connection negotiation failed: client connection to 127.9.176.254:46579: connect: Connection refused (error 111)
I20260808 22:22:21.894717 21147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:21.890005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37138 (local address 127.9.176.252:34903)
0808 22:22:21.890244 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:21.890248 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:21.890263 (+ 15us) server_negotiation.cc:407] Connection header received
0808 22:22:21.890851 (+ 588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:21.890854 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:21.890903 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:21.891001 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:22:21.891386 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:21.892166 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:21.893643 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:21.894267 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:21.894358 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:21.894417 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:21.894547 (+ 130us) server_negotiation.cc:299] Negotiation successful
0808 22:22:21.894625 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":96,"threads_started":1}
I20260808 22:22:21.894717 21145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:21.889852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.252:34903 (local address 127.0.0.1:37138)
0808 22:22:21.890142 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:21.890156 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:21.890253 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:21.891134 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:21.891137 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:21.891157 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:22:21.891258 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:21.891266 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:21.892267 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:21.892270 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:21.893535 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:21.893543 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:21.894367 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:21.894370 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:21.894439 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:21.894459 (+ 20us) client_negotiation.cc:769] Sending connection context
0808 22:22:21.894511 (+ 52us) client_negotiation.cc:240] Negotiation successful
0808 22:22:21.894572 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":56,"threads_started":1}
I20260808 22:22:21.895215 20345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "28f68f09769f4576ab963f9249cdf9e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "bacae03153b74596ba962b8db7fedc8f" is_pre_election: true
I20260808 22:22:21.895383 20345 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bacae03153b74596ba962b8db7fedc8f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28f68f09769f4576ab963f9249cdf9e1 in term 1.
I20260808 22:22:21.895697 20015 leader_election.cc:302] T 00000000000000000000000000000000 P 28f68f09769f4576ab963f9249cdf9e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28f68f09769f4576ab963f9249cdf9e1, bacae03153b74596ba962b8db7fedc8f; no voters: be929da42de644648042b1c514d26d16
I20260808 22:22:21.895867 21149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:21.890873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.253:37039 (local address 127.0.0.1:36126)
0808 22:22:21.891050 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:21.891062 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:21.891145 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:21.891993 (+ 848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:21.891996 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:21.892013 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:22:21.892119 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:21.892125 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:21.893345 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:21.893348 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:21.894495 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:21.894504 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:21.895479 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:21.895482 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:21.895571 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:21.895594 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 22:22:21.895686 (+ 92us) client_negotiation.cc:240] Negotiation successful
0808 22:22:21.895757 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":71,"threads_started":1}
I20260808 22:22:21.895874 21154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:21.891366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36126 (local address 127.9.176.253:37039)
0808 22:22:21.891788 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:21.891797 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:21.891810 (+ 13us) server_negotiation.cc:407] Connection header received
0808 22:22:21.891851 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:21.891853 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:21.891894 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:21.891972 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:22:21.892417 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:21.893245 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:21.894604 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:21.895375 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:21.895471 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:21.895525 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:21.895704 (+ 179us) server_negotiation.cc:299] Negotiation successful
0808 22:22:21.895765 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":97,"threads_started":1}
I20260808 22:22:21.895932 21140 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 28f68f09769f4576ab963f9249cdf9e1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 22:22:21.895980 21140 raft_consensus.cc:482] T 00000000000000000000000000000000 P 28f68f09769f4576ab963f9249cdf9e1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader be929da42de644648042b1c514d26d16)
I20260808 22:22:21.896026 21140 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 28f68f09769f4576ab963f9249cdf9e1 [term 1 FOLLOWER]: Advancing to term 2
I20260808 22:22:21.897060 21140 raft_consensus.cc:504] T 00000000000000000000000000000000 P 28f68f09769f4576ab963f9249cdf9e1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "be929da42de644648042b1c514d26d16" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 46579 } } peers { permanent_uuid: "28f68f09769f4576ab963f9249cdf9e1" member_type: VOTER last_known_addr { host: "127.9.176.253" port: 37039 } attrs { promote: false } } peers { permanent_uuid: "bacae03153b74596ba962b8db7fedc8f" member_type: VOTER last_known_addr { host: "127.9.176.252" port: 34903 } attrs { promote: false } }
I20260808 22:22:21.897282 20053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bacae03153b74596ba962b8db7fedc8f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "28f68f09769f4576ab963f9249cdf9e1" is_pre_election: true
I20260808 22:22:21.897360 21140 leader_election.cc:288] T 00000000000000000000000000000000 P 28f68f09769f4576ab963f9249cdf9e1 [CANDIDATE]: Term 2 election: Requested vote from peers be929da42de644648042b1c514d26d16 (127.9.176.254:46579), bacae03153b74596ba962b8db7fedc8f (127.9.176.252:34903)
I20260808 22:22:21.897490 20053 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 28f68f09769f4576ab963f9249cdf9e1 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bacae03153b74596ba962b8db7fedc8f in current term 2: Already voted for candidate 28f68f09769f4576ab963f9249cdf9e1 in this term.
I20260808 22:22:21.897650 20345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "28f68f09769f4576ab963f9249cdf9e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "bacae03153b74596ba962b8db7fedc8f"
I20260808 22:22:21.897728 21145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:21.897478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:46579 (local address 127.0.0.1:34612)
0808 22:22:21.897627 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:21.897675 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260808 22:22:21.897776 20345 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bacae03153b74596ba962b8db7fedc8f [term 1 FOLLOWER]: Advancing to term 2
I20260808 22:22:21.897886 20304 leader_election.cc:302] T 00000000000000000000000000000000 P bacae03153b74596ba962b8db7fedc8f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bacae03153b74596ba962b8db7fedc8f; no voters: 28f68f09769f4576ab963f9249cdf9e1, be929da42de644648042b1c514d26d16
I20260808 22:22:21.898347 21145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:21.898143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:46579 (local address 127.0.0.1:34620)
0808 22:22:21.898241 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:21.898293 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260808 22:22:21.898419 20013 leader_election.cc:334] T 00000000000000000000000000000000 P 28f68f09769f4576ab963f9249cdf9e1 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer be929da42de644648042b1c514d26d16 (127.9.176.254:46579): Network error: Client connection negotiation failed: client connection to 127.9.176.254:46579: connect: Connection refused (error 111)
I20260808 22:22:21.898844 20345 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bacae03153b74596ba962b8db7fedc8f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28f68f09769f4576ab963f9249cdf9e1 in term 2.
I20260808 22:22:21.898939 21142 raft_consensus.cc:2726] T 00000000000000000000000000000000 P bacae03153b74596ba962b8db7fedc8f [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 22:22:21.899144 20015 leader_election.cc:302] T 00000000000000000000000000000000 P 28f68f09769f4576ab963f9249cdf9e1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28f68f09769f4576ab963f9249cdf9e1, bacae03153b74596ba962b8db7fedc8f; no voters: be929da42de644648042b1c514d26d16
I20260808 22:22:21.899302 21140 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 28f68f09769f4576ab963f9249cdf9e1 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 22:22:21.899470 21140 raft_consensus.cc:686] T 00000000000000000000000000000000 P 28f68f09769f4576ab963f9249cdf9e1 [term 2 LEADER]: Becoming Leader. State: Replica: 28f68f09769f4576ab963f9249cdf9e1, State: Running, Role: LEADER
I20260808 22:22:21.899694 21140 consensus_queue.cc:237] T 00000000000000000000000000000000 P 28f68f09769f4576ab963f9249cdf9e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "be929da42de644648042b1c514d26d16" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 46579 } } peers { permanent_uuid: "28f68f09769f4576ab963f9249cdf9e1" member_type: VOTER last_known_addr { host: "127.9.176.253" port: 37039 } attrs { promote: false } } peers { permanent_uuid: "bacae03153b74596ba962b8db7fedc8f" member_type: VOTER last_known_addr { host: "127.9.176.252" port: 34903 } attrs { promote: false } }
I20260808 22:22:21.900604 21155 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28f68f09769f4576ab963f9249cdf9e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 28f68f09769f4576ab963f9249cdf9e1. Latest consensus state: current_term: 2 leader_uuid: "28f68f09769f4576ab963f9249cdf9e1" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "be929da42de644648042b1c514d26d16" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 46579 } } peers { permanent_uuid: "28f68f09769f4576ab963f9249cdf9e1" member_type: VOTER last_known_addr { host: "127.9.176.253" port: 37039 } attrs { promote: false } } peers { permanent_uuid: "bacae03153b74596ba962b8db7fedc8f" member_type: VOTER last_known_addr { host: "127.9.176.252" port: 34903 } attrs { promote: false } } }
I20260808 22:22:21.900694 21155 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28f68f09769f4576ab963f9249cdf9e1 [sys.catalog]: This master's current role is: LEADER
I20260808 22:22:21.900980 21157 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:22:21.903293 21157 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:22:21.903672 21157 catalog_manager.cc:1259] Loaded cluster ID: 68a7dc5d6ee2417c9d58328afa544c53
I20260808 22:22:21.903715 21157 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:22:21.904227 21157 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:22:21.904596 21157 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 28f68f09769f4576ab963f9249cdf9e1: Loaded TSK: 0
I20260808 22:22:21.905270 21157 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 22:22:21.939095 20035 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36118:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 22:22:21.942101 20035 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 22:22:21.944840 21145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:21.944608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:46579 (local address 127.0.0.1:34626)
0808 22:22:21.944711 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:21.944767 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260808 22:22:21.944957 20345 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bacae03153b74596ba962b8db7fedc8f [term 2 FOLLOWER]: Refusing update from remote peer 28f68f09769f4576ab963f9249cdf9e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260808 22:22:21.945410 21140 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 28f68f09769f4576ab963f9249cdf9e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bacae03153b74596ba962b8db7fedc8f" member_type: VOTER last_known_addr { host: "127.9.176.252" port: 34903 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260808 22:22:21.945756 21145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:21.945533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:46579 (local address 127.0.0.1:34634)
0808 22:22:21.945638 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:21.945693 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260808 22:22:21.945875 20013 consensus_peers.cc:487] T 00000000000000000000000000000000 P 28f68f09769f4576ab963f9249cdf9e1 -> Peer be929da42de644648042b1c514d26d16 (127.9.176.254:46579): Couldn't send request to peer be929da42de644648042b1c514d26d16. Status: Network error: Client connection negotiation failed: client connection to 127.9.176.254:46579: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260808 22:22:21.946890 21142 sys_catalog.cc:455] T 00000000000000000000000000000000 P bacae03153b74596ba962b8db7fedc8f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 28f68f09769f4576ab963f9249cdf9e1. Latest consensus state: current_term: 2 leader_uuid: "28f68f09769f4576ab963f9249cdf9e1" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "be929da42de644648042b1c514d26d16" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 46579 } } peers { permanent_uuid: "28f68f09769f4576ab963f9249cdf9e1" member_type: VOTER last_known_addr { host: "127.9.176.253" port: 37039 } attrs { promote: false } } peers { permanent_uuid: "bacae03153b74596ba962b8db7fedc8f" member_type: VOTER last_known_addr { host: "127.9.176.252" port: 34903 } attrs { promote: false } } }
I20260808 22:22:21.947002 21142 sys_catalog.cc:458] T 00000000000000000000000000000000 P bacae03153b74596ba962b8db7fedc8f [sys.catalog]: This master's current role is: FOLLOWER
I20260808 22:22:21.947930 21155 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28f68f09769f4576ab963f9249cdf9e1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "28f68f09769f4576ab963f9249cdf9e1" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "be929da42de644648042b1c514d26d16" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 46579 } } peers { permanent_uuid: "28f68f09769f4576ab963f9249cdf9e1" member_type: VOTER last_known_addr { host: "127.9.176.253" port: 37039 } attrs { promote: false } } peers { permanent_uuid: "bacae03153b74596ba962b8db7fedc8f" member_type: VOTER last_known_addr { host: "127.9.176.252" port: 34903 } attrs { promote: false } } }
I20260808 22:22:21.948040 21155 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28f68f09769f4576ab963f9249cdf9e1 [sys.catalog]: This master's current role is: LEADER
I20260808 22:22:21.948422 21142 sys_catalog.cc:455] T 00000000000000000000000000000000 P bacae03153b74596ba962b8db7fedc8f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "28f68f09769f4576ab963f9249cdf9e1" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "be929da42de644648042b1c514d26d16" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 46579 } } peers { permanent_uuid: "28f68f09769f4576ab963f9249cdf9e1" member_type: VOTER last_known_addr { host: "127.9.176.253" port: 37039 } attrs { promote: false } } peers { permanent_uuid: "bacae03153b74596ba962b8db7fedc8f" member_type: VOTER last_known_addr { host: "127.9.176.252" port: 34903 } attrs { promote: false } } }
I20260808 22:22:21.948531 21142 sys_catalog.cc:458] T 00000000000000000000000000000000 P bacae03153b74596ba962b8db7fedc8f [sys.catalog]: This master's current role is: FOLLOWER
I20260808 22:22:21.957887 21145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:21.953572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.194:46339 (local address 127.0.0.1:52350)
0808 22:22:21.953709 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:21.953722 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:21.953792 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:21.954781 (+ 989us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:21.954784 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:21.954802 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:22:21.954875 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:21.954881 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:21.955739 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:21.955741 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:21.956691 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:21.956701 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:21.957542 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:21.957546 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:21.957627 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:21.957656 (+ 29us) client_negotiation.cc:769] Sending connection context
0808 22:22:21.957707 (+ 51us) client_negotiation.cc:240] Negotiation successful
0808 22:22:21.957769 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79,"mutex_wait_us":82}
I20260808 22:22:21.958284 21170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:21.954031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.195:37359 (local address 127.0.0.1:59910)
0808 22:22:21.954701 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:21.954711 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:21.954790 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:21.955209 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:21.955212 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:21.955230 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:22:21.955305 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:21.955310 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:21.956186 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:21.956190 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:21.957192 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:21.957200 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:21.957929 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:21.957932 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:21.958018 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:21.958042 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 22:22:21.958098 (+ 56us) client_negotiation.cc:240] Negotiation successful
0808 22:22:21.958167 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"thread_start_us":91,"threads_started":1}
I20260808 22:22:21.958330 21174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:21.954319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59910 (local address 127.9.176.195:37359)
0808 22:22:21.954979 (+ 660us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:21.954983 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:21.954996 (+ 13us) server_negotiation.cc:407] Connection header received
0808 22:22:21.955056 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:21.955059 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:21.955104 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:21.955186 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:22:21.955405 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:21.956074 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:21.957302 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:21.957821 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:21.957917 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:21.958085 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:21.958134 (+ 49us) server_negotiation.cc:299] Negotiation successful
0808 22:22:21.958211 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":73,"threads_started":1}
I20260808 22:22:21.958567 21144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:21.953791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.193:34919 (local address 127.0.0.1:42822)
0808 22:22:21.953992 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:21.954006 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:21.954097 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:21.954525 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:21.954528 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:21.954547 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:22:21.954664 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:21.954671 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:21.955695 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:21.955698 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:21.956823 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:21.956833 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:21.958218 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:21.958222 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:21.958309 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:21.958330 (+ 21us) client_negotiation.cc:769] Sending connection context
0808 22:22:21.958379 (+ 49us) client_negotiation.cc:240] Negotiation successful
0808 22:22:21.958443 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":8}
I20260808 22:22:21.958850 21169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:21.953709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52350 (local address 127.9.176.194:46339)
0808 22:22:21.954335 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:21.954339 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:21.954353 (+ 14us) server_negotiation.cc:407] Connection header received
0808 22:22:21.954398 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:21.954401 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:21.954445 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:21.954522 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:22:21.954986 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:21.955618 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:21.956808 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:21.957420 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:21.958210 (+ 790us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:21.958272 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:21.958305 (+ 33us) server_negotiation.cc:299] Negotiation successful
0808 22:22:21.958725 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":63,"threads_started":1}
I20260808 22:22:21.958869 20936 tablet_service.cc:1467] Processing CreateTablet for tablet 2e2832409f114deebf710226d0db13de (DEFAULT_TABLE table=test-table [id=6fba71bf95cc4127a76e8fcc8bd25f33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:22:21.959111 21171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:21.954032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42822 (local address 127.9.176.193:34919)
0808 22:22:21.954275 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:21.954280 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:21.954294 (+ 14us) server_negotiation.cc:407] Connection header received
0808 22:22:21.954345 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:21.954348 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:21.954394 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:21.954511 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:22:21.954787 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:21.955590 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:21.957426 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:21.958092 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:21.958822 (+ 730us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:21.958884 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:21.958922 (+ 38us) server_negotiation.cc:299] Negotiation successful
0808 22:22:21.958992 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":87,"threads_started":1}
I20260808 22:22:21.959146 20936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e2832409f114deebf710226d0db13de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:22:21.959347 20723 tablet_service.cc:1467] Processing CreateTablet for tablet 2e2832409f114deebf710226d0db13de (DEFAULT_TABLE table=test-table [id=6fba71bf95cc4127a76e8fcc8bd25f33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:22:21.959525 20503 tablet_service.cc:1467] Processing CreateTablet for tablet 2e2832409f114deebf710226d0db13de (DEFAULT_TABLE table=test-table [id=6fba71bf95cc4127a76e8fcc8bd25f33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:22:21.959580 20723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e2832409f114deebf710226d0db13de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:22:21.959738 20503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e2832409f114deebf710226d0db13de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:22:21.961684 21175 tablet_bootstrap.cc:492] T 2e2832409f114deebf710226d0db13de P cf24739aa12a410fa19526e69e0c2d06: Bootstrap starting.
I20260808 22:22:21.962217 21177 tablet_bootstrap.cc:492] T 2e2832409f114deebf710226d0db13de P 39101d245faa4b6eb281ffc89ef69bf6: Bootstrap starting.
I20260808 22:22:21.962241 21176 tablet_bootstrap.cc:492] T 2e2832409f114deebf710226d0db13de P 41b49dbbd6a04fce85c78e62b80ba81e: Bootstrap starting.
I20260808 22:22:21.962713 21175 tablet_bootstrap.cc:654] T 2e2832409f114deebf710226d0db13de P cf24739aa12a410fa19526e69e0c2d06: Neither blocks nor log segments found. Creating new log.
I20260808 22:22:21.962960 21176 tablet_bootstrap.cc:654] T 2e2832409f114deebf710226d0db13de P 41b49dbbd6a04fce85c78e62b80ba81e: Neither blocks nor log segments found. Creating new log.
I20260808 22:22:21.963086 21177 tablet_bootstrap.cc:654] T 2e2832409f114deebf710226d0db13de P 39101d245faa4b6eb281ffc89ef69bf6: Neither blocks nor log segments found. Creating new log.
I20260808 22:22:21.963972 21175 tablet_bootstrap.cc:492] T 2e2832409f114deebf710226d0db13de P cf24739aa12a410fa19526e69e0c2d06: No bootstrap required, opened a new log
I20260808 22:22:21.964066 21175 ts_tablet_manager.cc:1397] T 2e2832409f114deebf710226d0db13de P cf24739aa12a410fa19526e69e0c2d06: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 22:22:21.964186 21177 tablet_bootstrap.cc:492] T 2e2832409f114deebf710226d0db13de P 39101d245faa4b6eb281ffc89ef69bf6: No bootstrap required, opened a new log
I20260808 22:22:21.964265 21177 ts_tablet_manager.cc:1397] T 2e2832409f114deebf710226d0db13de P 39101d245faa4b6eb281ffc89ef69bf6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 22:22:21.964509 21175 raft_consensus.cc:348] T 2e2832409f114deebf710226d0db13de P cf24739aa12a410fa19526e69e0c2d06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b49dbbd6a04fce85c78e62b80ba81e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 46339 } } peers { permanent_uuid: "39101d245faa4b6eb281ffc89ef69bf6" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34919 } } peers { permanent_uuid: "cf24739aa12a410fa19526e69e0c2d06" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37359 } }
I20260808 22:22:21.964636 21175 raft_consensus.cc:374] T 2e2832409f114deebf710226d0db13de P cf24739aa12a410fa19526e69e0c2d06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:22:21.964679 21175 raft_consensus.cc:729] T 2e2832409f114deebf710226d0db13de P cf24739aa12a410fa19526e69e0c2d06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf24739aa12a410fa19526e69e0c2d06, State: Initialized, Role: FOLLOWER
I20260808 22:22:21.964634 21177 raft_consensus.cc:348] T 2e2832409f114deebf710226d0db13de P 39101d245faa4b6eb281ffc89ef69bf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b49dbbd6a04fce85c78e62b80ba81e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 46339 } } peers { permanent_uuid: "39101d245faa4b6eb281ffc89ef69bf6" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34919 } } peers { permanent_uuid: "cf24739aa12a410fa19526e69e0c2d06" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37359 } }
I20260808 22:22:21.964725 21177 raft_consensus.cc:374] T 2e2832409f114deebf710226d0db13de P 39101d245faa4b6eb281ffc89ef69bf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:22:21.964761 21177 raft_consensus.cc:729] T 2e2832409f114deebf710226d0db13de P 39101d245faa4b6eb281ffc89ef69bf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39101d245faa4b6eb281ffc89ef69bf6, State: Initialized, Role: FOLLOWER
I20260808 22:22:21.964807 21175 consensus_queue.cc:260] T 2e2832409f114deebf710226d0db13de P cf24739aa12a410fa19526e69e0c2d06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b49dbbd6a04fce85c78e62b80ba81e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 46339 } } peers { permanent_uuid: "39101d245faa4b6eb281ffc89ef69bf6" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34919 } } peers { permanent_uuid: "cf24739aa12a410fa19526e69e0c2d06" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37359 } }
I20260808 22:22:21.964862 21177 consensus_queue.cc:260] T 2e2832409f114deebf710226d0db13de P 39101d245faa4b6eb281ffc89ef69bf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b49dbbd6a04fce85c78e62b80ba81e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 46339 } } peers { permanent_uuid: "39101d245faa4b6eb281ffc89ef69bf6" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34919 } } peers { permanent_uuid: "cf24739aa12a410fa19526e69e0c2d06" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37359 } }
I20260808 22:22:21.965157 21175 ts_tablet_manager.cc:1428] T 2e2832409f114deebf710226d0db13de P cf24739aa12a410fa19526e69e0c2d06: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 22:22:21.965188 21177 ts_tablet_manager.cc:1428] T 2e2832409f114deebf710226d0db13de P 39101d245faa4b6eb281ffc89ef69bf6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 22:22:21.965821 21187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:21.965441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:46579 (local address 127.9.176.195:40917)
0808 22:22:21.965670 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:21.965724 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":86,"threads_started":1}
I20260808 22:22:21.966014 21176 tablet_bootstrap.cc:492] T 2e2832409f114deebf710226d0db13de P 41b49dbbd6a04fce85c78e62b80ba81e: No bootstrap required, opened a new log
I20260808 22:22:21.966089 21176 ts_tablet_manager.cc:1397] T 2e2832409f114deebf710226d0db13de P 41b49dbbd6a04fce85c78e62b80ba81e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260808 22:22:21.966502 21176 raft_consensus.cc:348] T 2e2832409f114deebf710226d0db13de P 41b49dbbd6a04fce85c78e62b80ba81e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b49dbbd6a04fce85c78e62b80ba81e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 46339 } } peers { permanent_uuid: "39101d245faa4b6eb281ffc89ef69bf6" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34919 } } peers { permanent_uuid: "cf24739aa12a410fa19526e69e0c2d06" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37359 } }
I20260808 22:22:21.966614 21176 raft_consensus.cc:374] T 2e2832409f114deebf710226d0db13de P 41b49dbbd6a04fce85c78e62b80ba81e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:22:21.966645 21176 raft_consensus.cc:729] T 2e2832409f114deebf710226d0db13de P 41b49dbbd6a04fce85c78e62b80ba81e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41b49dbbd6a04fce85c78e62b80ba81e, State: Initialized, Role: FOLLOWER
I20260808 22:22:21.966751 21176 consensus_queue.cc:260] T 2e2832409f114deebf710226d0db13de P 41b49dbbd6a04fce85c78e62b80ba81e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b49dbbd6a04fce85c78e62b80ba81e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 46339 } } peers { permanent_uuid: "39101d245faa4b6eb281ffc89ef69bf6" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34919 } } peers { permanent_uuid: "cf24739aa12a410fa19526e69e0c2d06" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37359 } }
I20260808 22:22:21.967044 21176 ts_tablet_manager.cc:1428] T 2e2832409f114deebf710226d0db13de P 41b49dbbd6a04fce85c78e62b80ba81e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 22:22:21.968422 20621 heartbeater.cc:503] Master 127.9.176.253:37039 was elected leader, sending a full tablet report...
I20260808 22:22:21.969022 20826 heartbeater.cc:503] Master 127.9.176.253:37039 was elected leader, sending a full tablet report...
I20260808 22:22:21.969293 21189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:21.965533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:46579 (local address 127.9.176.193:34639)
0808 22:22:21.969168 (+ 3635us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:21.969214 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3573,"thread_start_us":92,"threads_started":1}
I20260808 22:22:21.970069 21189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:21.969682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:46579 (local address 127.9.176.193:50271)
0808 22:22:21.969944 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:21.969999 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":87,"threads_started":1}
I20260808 22:22:21.970592 21191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:21.967724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:46579 (local address 127.9.176.194:34549)
0808 22:22:21.970446 (+ 2722us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:21.970499 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2659,"thread_start_us":88,"threads_started":1}
I20260808 22:22:21.971020 21034 heartbeater.cc:503] Master 127.9.176.253:37039 was elected leader, sending a full tablet report...
I20260808 22:22:21.971230 21191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:21.971018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:46579 (local address 127.9.176.194:47697)
0808 22:22:21.971120 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:21.971170 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260808 22:22:21.971546 21187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:21.969917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:46579 (local address 127.9.176.195:34735)
0808 22:22:21.971432 (+ 1515us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:21.971484 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":747}
I20260808 22:22:21.981231 21186 raft_consensus.cc:482] T 2e2832409f114deebf710226d0db13de P 39101d245faa4b6eb281ffc89ef69bf6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 22:22:21.981313 21186 raft_consensus.cc:504] T 2e2832409f114deebf710226d0db13de P 39101d245faa4b6eb281ffc89ef69bf6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b49dbbd6a04fce85c78e62b80ba81e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 46339 } } peers { permanent_uuid: "39101d245faa4b6eb281ffc89ef69bf6" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34919 } } peers { permanent_uuid: "cf24739aa12a410fa19526e69e0c2d06" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37359 } }
I20260808 22:22:21.981638 21186 leader_election.cc:288] T 2e2832409f114deebf710226d0db13de P 39101d245faa4b6eb281ffc89ef69bf6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 41b49dbbd6a04fce85c78e62b80ba81e (127.9.176.194:46339), cf24739aa12a410fa19526e69e0c2d06 (127.9.176.195:37359)
I20260808 22:22:21.981921 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e2832409f114deebf710226d0db13de" candidate_uuid: "39101d245faa4b6eb281ffc89ef69bf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41b49dbbd6a04fce85c78e62b80ba81e" is_pre_election: true
I20260808 22:22:21.981948 20963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e2832409f114deebf710226d0db13de" candidate_uuid: "39101d245faa4b6eb281ffc89ef69bf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf24739aa12a410fa19526e69e0c2d06" is_pre_election: true
I20260808 22:22:21.982035 20750 raft_consensus.cc:2454] T 2e2832409f114deebf710226d0db13de P 41b49dbbd6a04fce85c78e62b80ba81e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39101d245faa4b6eb281ffc89ef69bf6 in term 0.
I20260808 22:22:21.982050 20963 raft_consensus.cc:2454] T 2e2832409f114deebf710226d0db13de P cf24739aa12a410fa19526e69e0c2d06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39101d245faa4b6eb281ffc89ef69bf6 in term 0.
I20260808 22:22:21.982335 20427 leader_election.cc:302] T 2e2832409f114deebf710226d0db13de P 39101d245faa4b6eb281ffc89ef69bf6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39101d245faa4b6eb281ffc89ef69bf6, 41b49dbbd6a04fce85c78e62b80ba81e; no voters:
I20260808 22:22:21.982481 21186 raft_consensus.cc:2781] T 2e2832409f114deebf710226d0db13de P 39101d245faa4b6eb281ffc89ef69bf6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 22:22:21.982537 21186 raft_consensus.cc:482] T 2e2832409f114deebf710226d0db13de P 39101d245faa4b6eb281ffc89ef69bf6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 22:22:21.982580 21186 raft_consensus.cc:3037] T 2e2832409f114deebf710226d0db13de P 39101d245faa4b6eb281ffc89ef69bf6 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:22:21.983564 21186 raft_consensus.cc:504] T 2e2832409f114deebf710226d0db13de P 39101d245faa4b6eb281ffc89ef69bf6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b49dbbd6a04fce85c78e62b80ba81e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 46339 } } peers { permanent_uuid: "39101d245faa4b6eb281ffc89ef69bf6" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34919 } } peers { permanent_uuid: "cf24739aa12a410fa19526e69e0c2d06" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37359 } }
I20260808 22:22:21.983855 21186 leader_election.cc:288] T 2e2832409f114deebf710226d0db13de P 39101d245faa4b6eb281ffc89ef69bf6 [CANDIDATE]: Term 1 election: Requested vote from peers 41b49dbbd6a04fce85c78e62b80ba81e (127.9.176.194:46339), cf24739aa12a410fa19526e69e0c2d06 (127.9.176.195:37359)
I20260808 22:22:21.984129 20750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e2832409f114deebf710226d0db13de" candidate_uuid: "39101d245faa4b6eb281ffc89ef69bf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41b49dbbd6a04fce85c78e62b80ba81e"
I20260808 22:22:21.984170 20963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e2832409f114deebf710226d0db13de" candidate_uuid: "39101d245faa4b6eb281ffc89ef69bf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf24739aa12a410fa19526e69e0c2d06"
I20260808 22:22:21.984223 20750 raft_consensus.cc:3037] T 2e2832409f114deebf710226d0db13de P 41b49dbbd6a04fce85c78e62b80ba81e [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:22:21.984268 20963 raft_consensus.cc:3037] T 2e2832409f114deebf710226d0db13de P cf24739aa12a410fa19526e69e0c2d06 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:22:21.985054 20750 raft_consensus.cc:2454] T 2e2832409f114deebf710226d0db13de P 41b49dbbd6a04fce85c78e62b80ba81e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39101d245faa4b6eb281ffc89ef69bf6 in term 1.
I20260808 22:22:21.985100 20963 raft_consensus.cc:2454] T 2e2832409f114deebf710226d0db13de P cf24739aa12a410fa19526e69e0c2d06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39101d245faa4b6eb281ffc89ef69bf6 in term 1.
I20260808 22:22:21.985297 20427 leader_election.cc:302] T 2e2832409f114deebf710226d0db13de P 39101d245faa4b6eb281ffc89ef69bf6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39101d245faa4b6eb281ffc89ef69bf6, 41b49dbbd6a04fce85c78e62b80ba81e; no voters:
I20260808 22:22:21.985525 21186 raft_consensus.cc:2781] T 2e2832409f114deebf710226d0db13de P 39101d245faa4b6eb281ffc89ef69bf6 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:22:21.985693 21186 raft_consensus.cc:686] T 2e2832409f114deebf710226d0db13de P 39101d245faa4b6eb281ffc89ef69bf6 [term 1 LEADER]: Becoming Leader. State: Replica: 39101d245faa4b6eb281ffc89ef69bf6, State: Running, Role: LEADER
I20260808 22:22:21.985809 21186 consensus_queue.cc:237] T 2e2832409f114deebf710226d0db13de P 39101d245faa4b6eb281ffc89ef69bf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b49dbbd6a04fce85c78e62b80ba81e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 46339 } } peers { permanent_uuid: "39101d245faa4b6eb281ffc89ef69bf6" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34919 } } peers { permanent_uuid: "cf24739aa12a410fa19526e69e0c2d06" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37359 } }
I20260808 22:22:21.986464 21192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:21.986211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:46579 (local address 127.9.176.193:42331)
0808 22:22:21.986331 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:21.986389 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260808 22:22:21.987121 20035 catalog_manager.cc:5495] T 2e2832409f114deebf710226d0db13de P 39101d245faa4b6eb281ffc89ef69bf6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39101d245faa4b6eb281ffc89ef69bf6 (127.9.176.193). New cstate: current_term: 1 leader_uuid: "39101d245faa4b6eb281ffc89ef69bf6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b49dbbd6a04fce85c78e62b80ba81e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 46339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39101d245faa4b6eb281ffc89ef69bf6" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf24739aa12a410fa19526e69e0c2d06" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37359 } health_report { overall_health: UNKNOWN } } }
I20260808 22:22:21.998347 21171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:21.995485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42824 (local address 127.9.176.193:34919)
0808 22:22:21.995588 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:21.995593 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:21.995605 (+ 12us) server_negotiation.cc:407] Connection header received
0808 22:22:21.995734 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:21.995737 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:21.995775 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:21.995833 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:22:21.996057 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:21.996553 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:21.997423 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:21.997635 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:21.997726 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:21.997958 (+ 232us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:22:21.998026 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:22:21.998082 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:21.998190 (+ 108us) server_negotiation.cc:299] Negotiation successful
0808 22:22:21.998240 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260808 22:22:21.999408 20827 tablet.cc:2369] T 2e2832409f114deebf710226d0db13de P 41b49dbbd6a04fce85c78e62b80ba81e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 22:22:22.000185 20750 raft_consensus.cc:1261] T 2e2832409f114deebf710226d0db13de P 41b49dbbd6a04fce85c78e62b80ba81e [term 1 FOLLOWER]: Refusing update from remote peer 39101d245faa4b6eb281ffc89ef69bf6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:22:22.000331 20963 raft_consensus.cc:1261] T 2e2832409f114deebf710226d0db13de P cf24739aa12a410fa19526e69e0c2d06 [term 1 FOLLOWER]: Refusing update from remote peer 39101d245faa4b6eb281ffc89ef69bf6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:22:22.000672 21186 consensus_queue.cc:1035] T 2e2832409f114deebf710226d0db13de P 39101d245faa4b6eb281ffc89ef69bf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41b49dbbd6a04fce85c78e62b80ba81e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 46339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:22:22.000830 21193 consensus_queue.cc:1035] T 2e2832409f114deebf710226d0db13de P 39101d245faa4b6eb281ffc89ef69bf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf24739aa12a410fa19526e69e0c2d06" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:22:22.002970 21187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:22.002650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:46579 (local address 127.9.176.195:45619)
0808 22:22:22.002848 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:22.002903 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
I20260808 22:22:22.004119 21191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:22.002257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:46579 (local address 127.9.176.194:52097)
0808 22:22:22.004017 (+ 1760us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:22.004069 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1692}
I20260808 22:22:22.005164 21192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:22.004883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:46579 (local address 127.9.176.193:41873)
0808 22:22:22.005045 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:22.005094 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260808 22:22:22.006614 21187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:22.006286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:46579 (local address 127.9.176.195:40671)
0808 22:22:22.006480 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:22.006551 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
I20260808 22:22:22.009428 21192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:22.009101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:46579 (local address 127.9.176.193:48457)
0808 22:22:22.009319 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:22.009367 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158}
I20260808 22:22:22.010445 21191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:22.010208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:46579 (local address 127.9.176.194:33483)
0808 22:22:22.010325 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:22.010380 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260808 22:22:22.010776 20035 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36118:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:22:22.010912 20035 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36118:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:22:22.014616 20035 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 28f68f09769f4576ab963f9249cdf9e1: Sending DeleteTablet for 3 replicas of tablet 2e2832409f114deebf710226d0db13de
I20260808 22:22:22.015188 20723 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e2832409f114deebf710226d0db13de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:22:22 UTC) from {username='oryx', principal='oryx/127.9.176.253@KRBTEST.COM'} at 127.0.0.1:52350
I20260808 22:22:22.015249 20936 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e2832409f114deebf710226d0db13de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:22:22 UTC) from {username='oryx', principal='oryx/127.9.176.253@KRBTEST.COM'} at 127.0.0.1:59910
I20260808 22:22:22.015481 20503 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e2832409f114deebf710226d0db13de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:22:22 UTC) from {username='oryx', principal='oryx/127.9.176.253@KRBTEST.COM'} at 127.0.0.1:42822
I20260808 22:22:22.015522 21208 tablet_replica.cc:330] T 2e2832409f114deebf710226d0db13de P 41b49dbbd6a04fce85c78e62b80ba81e: stopping tablet replica
I20260808 22:22:22.015631 21208 raft_consensus.cc:2229] T 2e2832409f114deebf710226d0db13de P 41b49dbbd6a04fce85c78e62b80ba81e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:22:22.015748 21208 raft_consensus.cc:2258] T 2e2832409f114deebf710226d0db13de P 41b49dbbd6a04fce85c78e62b80ba81e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:22:22.015807 21210 tablet_replica.cc:330] T 2e2832409f114deebf710226d0db13de P cf24739aa12a410fa19526e69e0c2d06: stopping tablet replica
I20260808 22:22:22.015923 21210 raft_consensus.cc:2229] T 2e2832409f114deebf710226d0db13de P cf24739aa12a410fa19526e69e0c2d06 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:22:22.016033 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 20266
I20260808 22:22:22.016039 21210 raft_consensus.cc:2258] T 2e2832409f114deebf710226d0db13de P cf24739aa12a410fa19526e69e0c2d06 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:22:22.016165 21211 tablet_replica.cc:330] T 2e2832409f114deebf710226d0db13de P 39101d245faa4b6eb281ffc89ef69bf6: stopping tablet replica
I20260808 22:22:22.016333 21211 raft_consensus.cc:2229] T 2e2832409f114deebf710226d0db13de P 39101d245faa4b6eb281ffc89ef69bf6 [term 1 LEADER]: Raft consensus shutting down.
I20260808 22:22:22.016474 21208 ts_tablet_manager.cc:1905] T 2e2832409f114deebf710226d0db13de P 41b49dbbd6a04fce85c78e62b80ba81e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:22:22.016677 21210 ts_tablet_manager.cc:1905] T 2e2832409f114deebf710226d0db13de P cf24739aa12a410fa19526e69e0c2d06: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:22:22.018589 21208 ts_tablet_manager.cc:1918] T 2e2832409f114deebf710226d0db13de P 41b49dbbd6a04fce85c78e62b80ba81e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 22:22:22.018661 21208 log.cc:1196] T 2e2832409f114deebf710226d0db13de P 41b49dbbd6a04fce85c78e62b80ba81e: Deleting WAL directory at /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-1/wal/wals/2e2832409f114deebf710226d0db13de
I20260808 22:22:22.018888 21208 ts_tablet_manager.cc:1939] T 2e2832409f114deebf710226d0db13de P 41b49dbbd6a04fce85c78e62b80ba81e: Deleting consensus metadata
I20260808 22:22:22.019397 21210 ts_tablet_manager.cc:1918] T 2e2832409f114deebf710226d0db13de P cf24739aa12a410fa19526e69e0c2d06: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 22:22:22.019482 21210 log.cc:1196] T 2e2832409f114deebf710226d0db13de P cf24739aa12a410fa19526e69e0c2d06: Deleting WAL directory at /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786227706282457-9923-0/minicluster-data/ts-2/wal/wals/2e2832409f114deebf710226d0db13de
I20260808 22:22:22.019471 20013 catalog_manager.cc:4841] TS 41b49dbbd6a04fce85c78e62b80ba81e (127.9.176.194:46339): tablet 2e2832409f114deebf710226d0db13de (table test-table [id=6fba71bf95cc4127a76e8fcc8bd25f33]) successfully deleted
I20260808 22:22:22.019761 21210 ts_tablet_manager.cc:1939] T 2e2832409f114deebf710226d0db13de P cf24739aa12a410fa19526e69e0c2d06: Deleting consensus metadata
I20260808 22:22:22.020417 20015 catalog_manager.cc:4841] TS cf24739aa12a410fa19526e69e0c2d06 (127.9.176.195:37359): tablet 2e2832409f114deebf710226d0db13de (table test-table [id=6fba71bf95cc4127a76e8fcc8bd25f33]) successfully deleted
I20260808 22:22:22.033041 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 20625
I20260808 22:22:22.033304 21144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:22.033069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.193:34919 (local address 127.0.0.1:42834)
0808 22:22:22.033181 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:22.033231 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.193:34919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260808 22:22:22.036954 20014 catalog_manager.cc:4584] TS 39101d245faa4b6eb281ffc89ef69bf6 (127.9.176.193:34919): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2e2832409f114deebf710226d0db13de: Network error: Client connection negotiation failed: client connection to 127.9.176.193:34919: connect: Connection refused (error 111)
I20260808 22:22:22.042846 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 20835
I20260808 22:22:22.052124 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 19816
I20260808 22:22:22.062026 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 19993
2026-08-08T22:22:22Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5283 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 22:22:22 dist-test-slave-ncp9 krb5kdc[21216](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 22:22:22 dist-test-slave-ncp9 krb5kdc[21216](info): set up 2 sockets
Aug 08 22:22:22 dist-test-slave-ncp9 krb5kdc[21216](info): commencing operation
krb5kdc: starting...
W20260808 22:22:23.179765 9923 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/krb5kdc/test-user.keytab.
Aug 08 22:22:23 dist-test-slave-ncp9 krb5kdc[21216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227743, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T22:22:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:22:23Z Disabled control of system clock
WARNING: no policy specified for oryx/127.9.176.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.176.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.176.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/krb5kdc/oryx_127.9.176.254.keytab.
Entry for principal oryx/127.9.176.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/krb5kdc/oryx_127.9.176.254.keytab.
I20260808 22:22:23.295717 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:46215
--webserver_interface=127.9.176.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/krb5kdc/oryx_127.9.176.254.keytab
--principal=oryx/127.9.176.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:40329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.176.254:46215
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:22:23.402318 21233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:22:23.402576 21233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:22:23.402616 21233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:22:23.406565 21233 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:22:23.406616 21233 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260808 22:22:23.406632 21233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:22:23.406648 21233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:22:23.406663 21233 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:22:23.406678 21233 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:22:23.411906 21233 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:40329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.176.254:46215
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/krb5kdc/oryx_127.9.176.254.keytab
--principal=oryx/127.9.176.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:46215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.176.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21233
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:22:23.412945 21233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:22:23.413954 21233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 22:22:23.422439 21233 server_base.cc:1034] running on GCE node
W20260808 22:22:23.422358 21239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:23.422364 21246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:23.422371 21243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:22:23.423177 21233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:22:23.424270 21233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:22:23.425426 21233 hybrid_clock.cc:648] HybridClock initialized: now 1786227743425412 us; error 24 us; skew 500 ppm
Aug 08 22:22:23 dist-test-slave-ncp9 krb5kdc[21216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227743, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.176.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:22:23.430912 21233 init.cc:377] Logged in from keytab as oryx/127.9.176.254@KRBTEST.COM (short username oryx)
I20260808 22:22:23.431785 21233 webserver.cc:460] Webserver started at http://127.9.176.254:37819/ using document root <none> and password file <none>
I20260808 22:22:23.432317 21233 fs_manager.cc:359] Metadata directory not provided
I20260808 22:22:23.432379 21233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:22:23.432588 21233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:22:23.434638 21233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/minicluster-data/master-0/data/instance:
uuid: "824baa547e754adfbe38c0b466a74c12"
format_stamp: "Formatted at 2026-08-08 22:22:23 on dist-test-slave-ncp9"
server_key: "3312e91b455e5ace6f4c63ceabaff2cb"
server_key_iv: "e30e735b8e56c34a701911178fe463b5"
server_key_version: "encryptionkey@0"
I20260808 22:22:23.435194 21233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/minicluster-data/master-0/wal/instance:
uuid: "824baa547e754adfbe38c0b466a74c12"
format_stamp: "Formatted at 2026-08-08 22:22:23 on dist-test-slave-ncp9"
server_key: "3312e91b455e5ace6f4c63ceabaff2cb"
server_key_iv: "e30e735b8e56c34a701911178fe463b5"
server_key_version: "encryptionkey@0"
I20260808 22:22:23.438862 21233 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 22:22:23.441300 21255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:23.442577 21233 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 22:22:23.442704 21233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/minicluster-data/master-0/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/minicluster-data/master-0/wal
uuid: "824baa547e754adfbe38c0b466a74c12"
format_stamp: "Formatted at 2026-08-08 22:22:23 on dist-test-slave-ncp9"
server_key: "3312e91b455e5ace6f4c63ceabaff2cb"
server_key_iv: "e30e735b8e56c34a701911178fe463b5"
server_key_version: "encryptionkey@0"
I20260808 22:22:23.442803 21233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:22:23.464660 21233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:22:23.467655 21233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:22:23.467837 21233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:22:23.475176 21233 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.254:46215
I20260808 22:22:23.475230 21368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.254:46215 every 8 connection(s)
I20260808 22:22:23.476054 21233 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
I20260808 22:22:23.478652 21369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:22:23.481184 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 21233
I20260808 22:22:23.481305 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/minicluster-data/master-0/wal/instance
I20260808 22:22:23.481570 9923 external_mini_cluster.cc:1428] Setting key 1938c3316f7470e4456649e48185d8e1
I20260808 22:22:23.484385 21369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 824baa547e754adfbe38c0b466a74c12: Bootstrap starting.
I20260808 22:22:23.486518 21369 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 824baa547e754adfbe38c0b466a74c12: Neither blocks nor log segments found. Creating new log.
I20260808 22:22:23.487367 21369 log.cc:824] T 00000000000000000000000000000000 P 824baa547e754adfbe38c0b466a74c12: Log is configured to *not* fsync() on all Append() calls
I20260808 22:22:23.489454 21369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 824baa547e754adfbe38c0b466a74c12: No bootstrap required, opened a new log
Aug 08 22:22:23 dist-test-slave-ncp9 krb5kdc[21216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227743, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM
I20260808 22:22:23.492760 21369 raft_consensus.cc:348] T 00000000000000000000000000000000 P 824baa547e754adfbe38c0b466a74c12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "824baa547e754adfbe38c0b466a74c12" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 46215 } }
I20260808 22:22:23.493016 21369 raft_consensus.cc:374] T 00000000000000000000000000000000 P 824baa547e754adfbe38c0b466a74c12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:22:23.493080 21369 raft_consensus.cc:729] T 00000000000000000000000000000000 P 824baa547e754adfbe38c0b466a74c12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 824baa547e754adfbe38c0b466a74c12, State: Initialized, Role: FOLLOWER
I20260808 22:22:23.493489 21369 consensus_queue.cc:260] T 00000000000000000000000000000000 P 824baa547e754adfbe38c0b466a74c12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "824baa547e754adfbe38c0b466a74c12" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 46215 } }
I20260808 22:22:23.493613 21369 raft_consensus.cc:388] T 00000000000000000000000000000000 P 824baa547e754adfbe38c0b466a74c12 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:22:23.493670 21369 raft_consensus.cc:482] T 00000000000000000000000000000000 P 824baa547e754adfbe38c0b466a74c12 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:22:23.493764 21369 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 824baa547e754adfbe38c0b466a74c12 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:22:23.494638 21369 raft_consensus.cc:504] T 00000000000000000000000000000000 P 824baa547e754adfbe38c0b466a74c12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "824baa547e754adfbe38c0b466a74c12" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 46215 } }
I20260808 22:22:23.494944 21369 leader_election.cc:302] T 00000000000000000000000000000000 P 824baa547e754adfbe38c0b466a74c12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 824baa547e754adfbe38c0b466a74c12; no voters:
I20260808 22:22:23.495239 21369 leader_election.cc:288] T 00000000000000000000000000000000 P 824baa547e754adfbe38c0b466a74c12 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:22:23.495257 21373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:23.482979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58502 (local address 127.9.176.254:46215)
0808 22:22:23.483387 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:23.483402 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:23.483435 (+ 33us) server_negotiation.cc:407] Connection header received
0808 22:22:23.484037 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:23.484056 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:23.484380 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:23.484705 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:23.485524 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:23.486508 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:23.487670 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:23.488125 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:23.488234 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:23.490862 (+ 2628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:23.490866 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:23.490878 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:22:23.490903 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:23.492461 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:23.492857 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:23.492862 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:23.492866 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:23.493013 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:23.493269 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:23.493272 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:23.493274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:23.493734 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:23.493936 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:23.494201 (+ 265us) server_negotiation.cc:299] Negotiation successful
0808 22:22:23.494474 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":116,"threads_started":1}
I20260808 22:22:23.495322 21377 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 824baa547e754adfbe38c0b466a74c12 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:22:23.495496 21377 raft_consensus.cc:686] T 00000000000000000000000000000000 P 824baa547e754adfbe38c0b466a74c12 [term 1 LEADER]: Becoming Leader. State: Replica: 824baa547e754adfbe38c0b466a74c12, State: Running, Role: LEADER
I20260808 22:22:23.495783 21377 consensus_queue.cc:237] T 00000000000000000000000000000000 P 824baa547e754adfbe38c0b466a74c12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "824baa547e754adfbe38c0b466a74c12" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 46215 } }
I20260808 22:22:23.496464 21369 sys_catalog.cc:564] T 00000000000000000000000000000000 P 824baa547e754adfbe38c0b466a74c12 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:22:23.497931 21379 sys_catalog.cc:455] T 00000000000000000000000000000000 P 824baa547e754adfbe38c0b466a74c12 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 824baa547e754adfbe38c0b466a74c12. Latest consensus state: current_term: 1 leader_uuid: "824baa547e754adfbe38c0b466a74c12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "824baa547e754adfbe38c0b466a74c12" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 46215 } } }
I20260808 22:22:23.497962 21378 sys_catalog.cc:455] T 00000000000000000000000000000000 P 824baa547e754adfbe38c0b466a74c12 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "824baa547e754adfbe38c0b466a74c12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "824baa547e754adfbe38c0b466a74c12" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 46215 } } }
I20260808 22:22:23.498054 21379 sys_catalog.cc:458] T 00000000000000000000000000000000 P 824baa547e754adfbe38c0b466a74c12 [sys.catalog]: This master's current role is: LEADER
I20260808 22:22:23.498065 21378 sys_catalog.cc:458] T 00000000000000000000000000000000 P 824baa547e754adfbe38c0b466a74c12 [sys.catalog]: This master's current role is: LEADER
I20260808 22:22:23.498497 21398 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:22:23.501845 21398 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:22:23.506626 21398 catalog_manager.cc:1347] Generated new cluster ID: 7ac8c51cdc4a48a8bf527eef0eaccd54
I20260808 22:22:23.506690 21398 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:22:23.515653 21398 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:22:23.516479 21398 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:22:23.521598 21398 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 824baa547e754adfbe38c0b466a74c12: Generated new TSK 0
I20260808 22:22:23.522244 21398 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.9.176.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.176.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.176.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/krb5kdc/oryx_127.9.176.193.keytab.
Entry for principal oryx/127.9.176.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/krb5kdc/oryx_127.9.176.193.keytab.
I20260808 22:22:23.564750 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.193:0
--local_ip_for_outbound_sockets=127.9.176.193
--webserver_interface=127.9.176.193
--webserver_port=0
--tserver_master_addrs=127.9.176.254:46215
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/krb5kdc/oryx_127.9.176.193.keytab
--principal=oryx/127.9.176.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:40329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:22:23.677390 21414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:22:23.677634 21414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:22:23.677675 21414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:22:23.681592 21414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:22:23.681648 21414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:22:23.681691 21414 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 22:22:23.681742 21414 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.193
I20260808 22:22:23.686837 21414 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:40329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/krb5kdc/oryx_127.9.176.193.keytab
--principal=oryx/127.9.176.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.176.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.176.254:46215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21414
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.193
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:22:23.687835 21414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:22:23.688828 21414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:22:23.698146 21424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:23.698187 21423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:23.698230 21432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:22:23.698537 21414 server_base.cc:1034] running on GCE node
I20260808 22:22:23.698938 21414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:22:23.699561 21414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:22:23.700714 21414 hybrid_clock.cc:648] HybridClock initialized: now 1786227743700689 us; error 42 us; skew 500 ppm
Aug 08 22:22:23 dist-test-slave-ncp9 krb5kdc[21216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227743, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.176.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:22:23.703466 21414 init.cc:377] Logged in from keytab as oryx/127.9.176.193@KRBTEST.COM (short username oryx)
I20260808 22:22:23.704339 21414 webserver.cc:460] Webserver started at http://127.9.176.193:39285/ using document root <none> and password file <none>
I20260808 22:22:23.704877 21414 fs_manager.cc:359] Metadata directory not provided
I20260808 22:22:23.704988 21414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:22:23.705207 21414 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:22:23.707247 21414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/minicluster-data/ts-0/data/instance:
uuid: "7b49d3ab4f0c4429907d17e534a82add"
format_stamp: "Formatted at 2026-08-08 22:22:23 on dist-test-slave-ncp9"
server_key: "2bf17421ffaf8a8ba99d3c0837f6a453"
server_key_iv: "c93cba602182d6f1589fcdd590fc9f90"
server_key_version: "encryptionkey@0"
I20260808 22:22:23.707809 21414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/minicluster-data/ts-0/wal/instance:
uuid: "7b49d3ab4f0c4429907d17e534a82add"
format_stamp: "Formatted at 2026-08-08 22:22:23 on dist-test-slave-ncp9"
server_key: "2bf17421ffaf8a8ba99d3c0837f6a453"
server_key_iv: "c93cba602182d6f1589fcdd590fc9f90"
server_key_version: "encryptionkey@0"
I20260808 22:22:23.711493 21414 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260808 22:22:23.713882 21449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:23.715035 21414 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 22:22:23.715142 21414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/minicluster-data/ts-0/wal
uuid: "7b49d3ab4f0c4429907d17e534a82add"
format_stamp: "Formatted at 2026-08-08 22:22:23 on dist-test-slave-ncp9"
server_key: "2bf17421ffaf8a8ba99d3c0837f6a453"
server_key_iv: "c93cba602182d6f1589fcdd590fc9f90"
server_key_version: "encryptionkey@0"
I20260808 22:22:23.715240 21414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:22:23.728302 21414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:22:23.730990 21414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:22:23.731169 21414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:22:23.732540 21414 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:22:23.732589 21414 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:23.732640 21414 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:22:23.732676 21414 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 22:22:23 dist-test-slave-ncp9 krb5kdc[21216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227743, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.176.193@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM
I20260808 22:22:23.742743 21414 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.193:44137
I20260808 22:22:23.742869 21617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.193:44137 every 8 connection(s)
I20260808 22:22:23.743723 21414 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
I20260808 22:22:23.746840 21373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:23.733415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.193:38213 (local address 127.9.176.254:46215)
0808 22:22:23.733566 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:23.733570 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:23.734527 (+ 957us) server_negotiation.cc:407] Connection header received
0808 22:22:23.735561 (+ 1034us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:23.735565 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:23.735627 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:23.735690 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:23.737462 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:23.738297 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:23.739996 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:23.740372 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:23.740492 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:23.743336 (+ 2844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:23.743340 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:23.743342 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:22:23.743367 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:23.744837 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:23.745323 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:23.745327 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:23.745329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:23.745409 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:23.745716 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:23.745720 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:23.745721 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:23.745934 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:23.746021 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:23.746599 (+ 578us) server_negotiation.cc:299] Negotiation successful
0808 22:22:23.746721 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260808 22:22:23.747635 21464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:23.733876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:46215 (local address 127.9.176.193:38213)
0808 22:22:23.734355 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:23.734405 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:23.735276 (+ 871us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:23.736134 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:23.736144 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:23.736583 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:23.737290 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:23.737312 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:23.738470 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:23.738474 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:23.739852 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:23.739865 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:23.740538 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:23.740541 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:23.740711 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:23.741414 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:23.741438 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:23.743144 (+ 1706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:22:23.744955 (+ 1811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:23.744961 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:23.744975 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:23.745228 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:23.745491 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:23.745494 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:23.745496 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:23.745625 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:23.746034 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:22:23.746040 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:22:23.746332 (+ 292us) client_negotiation.cc:769] Sending connection context
0808 22:22:23.746517 (+ 185us) client_negotiation.cc:240] Negotiation successful
0808 22:22:23.746800 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":126,"threads_started":1}
I20260808 22:22:23.748765 21621 heartbeater.cc:347] Connected to a master server at 127.9.176.254:46215
I20260808 22:22:23.748930 21621 heartbeater.cc:464] Registering TS with master...
I20260808 22:22:23.749464 21621 heartbeater.cc:511] Master 127.9.176.254:46215 requested a full tablet report, sending...
I20260808 22:22:23.751048 21287 ts_manager.cc:194] Registered new tserver with Master: 7b49d3ab4f0c4429907d17e534a82add (127.9.176.193:44137)
I20260808 22:22:23.752189 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 21414
I20260808 22:22:23.752306 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/minicluster-data/ts-0/wal/instance
I20260808 22:22:23.752509 21287 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.176.193@KRBTEST.COM'} at 127.9.176.193:38213
I20260808 22:22:23.752599 9923 external_mini_cluster.cc:1428] Setting key 01db5e0bd585a0a183b716221ddc8e79
I20260808 22:22:23.757922 21373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:23.750970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.193:57049 (local address 127.9.176.254:46215)
0808 22:22:23.751082 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:23.751086 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:23.751238 (+ 152us) server_negotiation.cc:407] Connection header received
0808 22:22:23.751346 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:23.751349 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:23.751390 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:23.751464 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:23.751834 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:23.752581 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:23.753705 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:23.754011 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:23.754132 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:23.755035 (+ 903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:23.755038 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:23.755041 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:22:23.755072 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:23.756418 (+ 1346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:23.756868 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:23.756871 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:23.756873 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:23.756985 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:23.757264 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:23.757266 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:23.757267 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:23.757420 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:23.757489 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:23.757737 (+ 248us) server_negotiation.cc:299] Negotiation successful
0808 22:22:23.757821 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260808 22:22:23.757980 21632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:23.750913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:46215 (local address 127.9.176.193:57049)
0808 22:22:23.751165 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:23.751177 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:23.751266 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:23.751476 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:23.751478 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:23.751644 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:23.751732 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:23.751737 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:23.752730 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:23.752733 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:23.753578 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:23.753589 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:23.754134 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:23.754137 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:23.754216 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:23.754620 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:23.754629 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:23.754907 (+ 278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:22:23.756534 (+ 1627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:23.756538 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:23.756540 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:23.756765 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:23.757078 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:23.757081 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:23.757083 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:23.757184 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:23.757503 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:22:23.757506 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:22:23.757594 (+ 88us) client_negotiation.cc:769] Sending connection context
0808 22:22:23.757690 (+ 96us) client_negotiation.cc:240] Negotiation successful
0808 22:22:23.757812 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":93,"threads_started":1}
WARNING: no policy specified for oryx/127.9.176.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.176.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.176.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/krb5kdc/oryx_127.9.176.194.keytab.
Entry for principal oryx/127.9.176.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/krb5kdc/oryx_127.9.176.194.keytab.
I20260808 22:22:23.779501 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.194:0
--local_ip_for_outbound_sockets=127.9.176.194
--webserver_interface=127.9.176.194
--webserver_port=0
--tserver_master_addrs=127.9.176.254:46215
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/krb5kdc/oryx_127.9.176.194.keytab
--principal=oryx/127.9.176.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:40329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:22:23.887398 21636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:22:23.887650 21636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:22:23.887708 21636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:22:23.891623 21636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:22:23.891677 21636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:22:23.891728 21636 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 22:22:23.891780 21636 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.194
I20260808 22:22:23.896852 21636 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:40329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/krb5kdc/oryx_127.9.176.194.keytab
--principal=oryx/127.9.176.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.176.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.176.254:46215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21636
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.194
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:22:23.897863 21636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:22:23.898878 21636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:22:23.908052 21646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:23.908098 21650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:22:23.908182 21636 server_base.cc:1034] running on GCE node
W20260808 22:22:23.908092 21648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:22:23.908690 21636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:22:23.909297 21636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:22:23.910450 21636 hybrid_clock.cc:648] HybridClock initialized: now 1786227743910441 us; error 36 us; skew 500 ppm
Aug 08 22:22:23 dist-test-slave-ncp9 krb5kdc[21216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227743, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.176.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:22:23.913022 21636 init.cc:377] Logged in from keytab as oryx/127.9.176.194@KRBTEST.COM (short username oryx)
I20260808 22:22:23.913803 21636 webserver.cc:460] Webserver started at http://127.9.176.194:41551/ using document root <none> and password file <none>
I20260808 22:22:23.914325 21636 fs_manager.cc:359] Metadata directory not provided
I20260808 22:22:23.914390 21636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:22:23.914587 21636 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:22:23.916486 21636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/minicluster-data/ts-1/data/instance:
uuid: "3e22dfd749ff4f02b4c9ac55e651447e"
format_stamp: "Formatted at 2026-08-08 22:22:23 on dist-test-slave-ncp9"
server_key: "cd9d7a40c0b154b0ca1cac8e3ba88fd7"
server_key_iv: "52d043fa137e5cfac89a8b46f96e20f3"
server_key_version: "encryptionkey@0"
I20260808 22:22:23.917055 21636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/minicluster-data/ts-1/wal/instance:
uuid: "3e22dfd749ff4f02b4c9ac55e651447e"
format_stamp: "Formatted at 2026-08-08 22:22:23 on dist-test-slave-ncp9"
server_key: "cd9d7a40c0b154b0ca1cac8e3ba88fd7"
server_key_iv: "52d043fa137e5cfac89a8b46f96e20f3"
server_key_version: "encryptionkey@0"
I20260808 22:22:23.920616 21636 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260808 22:22:23.922880 21658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:23.923987 21636 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 22:22:23.924093 21636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/minicluster-data/ts-1/wal
uuid: "3e22dfd749ff4f02b4c9ac55e651447e"
format_stamp: "Formatted at 2026-08-08 22:22:23 on dist-test-slave-ncp9"
server_key: "cd9d7a40c0b154b0ca1cac8e3ba88fd7"
server_key_iv: "52d043fa137e5cfac89a8b46f96e20f3"
server_key_version: "encryptionkey@0"
I20260808 22:22:23.924180 21636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:22:23.955832 21636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:22:23.958613 21636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:22:23.958786 21636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:22:23.960256 21636 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:22:23.960307 21636 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:23.960361 21636 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:22:23.960386 21636 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 22:22:23 dist-test-slave-ncp9 krb5kdc[21216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227743, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.176.194@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM
I20260808 22:22:23.970690 21636 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.194:36787
I20260808 22:22:23.970746 21848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.194:36787 every 8 connection(s)
I20260808 22:22:23.971752 21636 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
I20260808 22:22:23.973969 21373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:23.960958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.194:58319 (local address 127.9.176.254:46215)
0808 22:22:23.961115 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:23.961119 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:23.961987 (+ 868us) server_negotiation.cc:407] Connection header received
0808 22:22:23.963230 (+ 1243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:23.963234 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:23.963296 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:23.963355 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:23.965175 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:23.965802 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:23.967489 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:23.967727 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:23.967826 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:23.970390 (+ 2564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:23.970394 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:23.970397 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:22:23.970428 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:23.971928 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:23.972482 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:23.972485 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:23.972487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:23.972586 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:23.972927 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:23.972930 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:23.972932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:23.973111 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:23.973197 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:23.973689 (+ 492us) server_negotiation.cc:299] Negotiation successful
0808 22:22:23.973811 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260808 22:22:23.974682 21670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:23.961332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:46215 (local address 127.9.176.194:58319)
0808 22:22:23.961808 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:23.961865 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:23.962952 (+ 1087us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:23.963801 (+ 849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:23.963811 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:23.964276 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:23.965000 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:23.965024 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:23.965918 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:23.965921 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:23.967347 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:23.967360 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:23.967879 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:23.967884 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:23.968056 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:23.968660 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:23.968679 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:23.970211 (+ 1532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:22:23.972034 (+ 1823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:23.972038 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:23.972050 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:23.972357 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:23.972674 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:23.972678 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:23.972680 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:23.972810 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:23.973218 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:22:23.973225 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:22:23.973443 (+ 218us) client_negotiation.cc:769] Sending connection context
0808 22:22:23.973594 (+ 151us) client_negotiation.cc:240] Negotiation successful
0808 22:22:23.973910 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":114,"threads_started":1}
I20260808 22:22:23.975276 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 21636
I20260808 22:22:23.975530 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/minicluster-data/ts-1/wal/instance
I20260808 22:22:23.975853 9923 external_mini_cluster.cc:1428] Setting key e7b7506aea9b7e9ae03686a41182a5fd
I20260808 22:22:23.976064 21852 heartbeater.cc:347] Connected to a master server at 127.9.176.254:46215
I20260808 22:22:23.976212 21852 heartbeater.cc:464] Registering TS with master...
I20260808 22:22:23.976737 21852 heartbeater.cc:511] Master 127.9.176.254:46215 requested a full tablet report, sending...
I20260808 22:22:23.977799 21287 ts_manager.cc:194] Registered new tserver with Master: 3e22dfd749ff4f02b4c9ac55e651447e (127.9.176.194:36787)
I20260808 22:22:23.978431 21287 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.176.194@KRBTEST.COM'} at 127.9.176.194:58319
I20260808 22:22:23.985646 21373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:23.978954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.194:39547 (local address 127.9.176.254:46215)
0808 22:22:23.979078 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:23.979083 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:23.979346 (+ 263us) server_negotiation.cc:407] Connection header received
0808 22:22:23.979464 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:23.979467 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:23.979511 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:23.979578 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:23.980014 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:23.980512 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:23.981377 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:23.981610 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:23.981701 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:23.982729 (+ 1028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:23.982733 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:23.982736 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:22:23.982762 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:23.984103 (+ 1341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:23.984514 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:23.984517 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:23.984519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:23.984594 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:23.984837 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:23.984840 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:23.984842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:23.985075 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:23.985163 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:23.985404 (+ 241us) server_negotiation.cc:299] Negotiation successful
0808 22:22:23.985511 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 22:22:23.985654 21863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:23.978890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:46215 (local address 127.9.176.194:39547)
0808 22:22:23.979261 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:23.979273 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:23.979367 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:23.979606 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:23.979609 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:23.979797 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:23.979893 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:23.979899 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:23.980636 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:23.980640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:23.981264 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:23.981274 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:23.981726 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:23.981730 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:23.981822 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:23.982279 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:23.982290 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:23.982596 (+ 306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:22:23.984215 (+ 1619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:23.984218 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:23.984220 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:23.984404 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:23.984676 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:23.984679 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:23.984681 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:23.984758 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:23.985165 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:22:23.985168 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:22:23.985258 (+ 90us) client_negotiation.cc:769] Sending connection context
0808 22:22:23.985342 (+ 84us) client_negotiation.cc:240] Negotiation successful
0808 22:22:23.985468 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":120,"threads_started":1}
WARNING: no policy specified for oryx/127.9.176.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.176.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.176.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/krb5kdc/oryx_127.9.176.195.keytab.
Entry for principal oryx/127.9.176.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/krb5kdc/oryx_127.9.176.195.keytab.
I20260808 22:22:24.003201 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.195:0
--local_ip_for_outbound_sockets=127.9.176.195
--webserver_interface=127.9.176.195
--webserver_port=0
--tserver_master_addrs=127.9.176.254:46215
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/krb5kdc/oryx_127.9.176.195.keytab
--principal=oryx/127.9.176.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:40329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:22:24.112753 21865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:22:24.113023 21865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:22:24.113078 21865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:22:24.116977 21865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:22:24.117026 21865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:22:24.117071 21865 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 22:22:24.117123 21865 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.195
I20260808 22:22:24.122244 21865 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:40329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/krb5kdc/oryx_127.9.176.195.keytab
--principal=oryx/127.9.176.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.176.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.176.254:46215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21865
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.195
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:22:24.123180 21865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:22:24.124140 21865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:22:24.133210 21876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:24.133285 21871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:24.133201 21872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:22:24.133478 21865 server_base.cc:1034] running on GCE node
I20260808 22:22:24.133852 21865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:22:24.134439 21865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:22:24.135593 21865 hybrid_clock.cc:648] HybridClock initialized: now 1786227744135578 us; error 28 us; skew 500 ppm
Aug 08 22:22:24 dist-test-slave-ncp9 krb5kdc[21216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227744, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.176.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:22:24.138146 21865 init.cc:377] Logged in from keytab as oryx/127.9.176.195@KRBTEST.COM (short username oryx)
I20260808 22:22:24.138998 21865 webserver.cc:460] Webserver started at http://127.9.176.195:36791/ using document root <none> and password file <none>
I20260808 22:22:24.139505 21865 fs_manager.cc:359] Metadata directory not provided
I20260808 22:22:24.139571 21865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:22:24.139765 21865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:22:24.141741 21865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/minicluster-data/ts-2/data/instance:
uuid: "522121848ad548a49cf2f2ee25898ec3"
format_stamp: "Formatted at 2026-08-08 22:22:24 on dist-test-slave-ncp9"
server_key: "8651b3dd5239b9f4063a19c743a60a6e"
server_key_iv: "01c517c8ccd5152fcef9ed2b073ba66d"
server_key_version: "encryptionkey@0"
I20260808 22:22:24.142292 21865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/minicluster-data/ts-2/wal/instance:
uuid: "522121848ad548a49cf2f2ee25898ec3"
format_stamp: "Formatted at 2026-08-08 22:22:24 on dist-test-slave-ncp9"
server_key: "8651b3dd5239b9f4063a19c743a60a6e"
server_key_iv: "01c517c8ccd5152fcef9ed2b073ba66d"
server_key_version: "encryptionkey@0"
I20260808 22:22:24.145704 21865 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 22:22:24.147897 21888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:24.149084 21865 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 22:22:24.149194 21865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/minicluster-data/ts-2/wal
uuid: "522121848ad548a49cf2f2ee25898ec3"
format_stamp: "Formatted at 2026-08-08 22:22:24 on dist-test-slave-ncp9"
server_key: "8651b3dd5239b9f4063a19c743a60a6e"
server_key_iv: "01c517c8ccd5152fcef9ed2b073ba66d"
server_key_version: "encryptionkey@0"
I20260808 22:22:24.149278 21865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:22:24.161449 21865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:22:24.163805 21865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:22:24.163956 21865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:22:24.165297 21865 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:22:24.165349 21865 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:24.165396 21865 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:22:24.165423 21865 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 22:22:24 dist-test-slave-ncp9 krb5kdc[21216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227744, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.176.195@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM
I20260808 22:22:24.175661 21865 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.195:39019
I20260808 22:22:24.175812 22076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.195:39019 every 8 connection(s)
I20260808 22:22:24.176523 21865 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
I20260808 22:22:24.178419 21373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:24.165894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.195:59251 (local address 127.9.176.254:46215)
0808 22:22:24.166010 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:24.166015 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:24.166952 (+ 937us) server_negotiation.cc:407] Connection header received
0808 22:22:24.167805 (+ 853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:24.167809 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:24.167870 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:24.167954 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:24.169459 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:24.170131 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:24.171775 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:24.172093 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:24.172194 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:24.174762 (+ 2568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:24.174766 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:24.174769 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:22:24.174795 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:24.176360 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:24.176912 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:24.176916 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:24.176918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:24.177017 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:24.177400 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:24.177403 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:24.177405 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:24.177598 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:24.177687 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:24.178137 (+ 450us) server_negotiation.cc:299] Negotiation successful
0808 22:22:24.178260 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260808 22:22:24.178973 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 21865
I20260808 22:22:24.179088 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/minicluster-data/ts-2/wal/instance
I20260808 22:22:24.179123 21907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:24.166362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:46215 (local address 127.9.176.195:59251)
0808 22:22:24.166808 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:24.166846 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:24.167608 (+ 762us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:24.168163 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:24.168172 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:24.168619 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:24.169290 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:24.169310 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:24.170244 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:24.170248 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:24.171633 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:24.171645 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:24.172262 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:24.172267 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:24.172431 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:24.173008 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:24.173023 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:24.174602 (+ 1579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:22:24.176467 (+ 1865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:24.176472 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:24.176486 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:24.176808 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:24.177136 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:24.177139 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:24.177141 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:24.177306 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:24.177700 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:22:24.177706 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:22:24.177913 (+ 207us) client_negotiation.cc:769] Sending connection context
0808 22:22:24.178077 (+ 164us) client_negotiation.cc:240] Negotiation successful
0808 22:22:24.178361 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":122,"threads_started":1}
I20260808 22:22:24.179425 9923 external_mini_cluster.cc:1428] Setting key ac7b99f7781393de2c1033ed698c2044
I20260808 22:22:24.180370 22077 heartbeater.cc:347] Connected to a master server at 127.9.176.254:46215
I20260808 22:22:24.180675 22077 heartbeater.cc:464] Registering TS with master...
I20260808 22:22:24.181254 22077 heartbeater.cc:511] Master 127.9.176.254:46215 requested a full tablet report, sending...
I20260808 22:22:24.182286 21288 ts_manager.cc:194] Registered new tserver with Master: 522121848ad548a49cf2f2ee25898ec3 (127.9.176.195:39019)
I20260808 22:22:24.182888 21288 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.176.195@KRBTEST.COM'} at 127.9.176.195:59251
I20260808 22:22:24.188397 22091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:24.182432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:46215 (local address 127.9.176.195:55321)
0808 22:22:24.182747 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:24.182759 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:24.182845 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:24.183126 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:24.183129 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:24.183279 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:24.183365 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:24.183370 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:24.184094 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:24.184097 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:24.184646 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:24.184653 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:24.185124 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:24.185128 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:24.185217 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:24.185574 (+ 357us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:24.185584 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:24.185845 (+ 261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:22:24.187077 (+ 1232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:24.187080 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:24.187082 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:24.187256 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:24.187502 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:24.187505 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:24.187507 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:24.187583 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:24.187919 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:22:24.187922 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:22:24.188007 (+ 85us) client_negotiation.cc:769] Sending connection context
0808 22:22:24.188101 (+ 94us) client_negotiation.cc:240] Negotiation successful
0808 22:22:24.188230 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":77,"threads_started":1}
I20260808 22:22:24.188395 21373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:24.182501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.195:55321 (local address 127.9.176.254:46215)
0808 22:22:24.182624 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:24.182628 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:24.182824 (+ 196us) server_negotiation.cc:407] Connection header received
0808 22:22:24.182914 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:24.182917 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:24.182955 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:24.183017 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:24.183475 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:24.183963 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:24.184752 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:24.185026 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:24.185114 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:24.185969 (+ 855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:24.185973 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:24.185975 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:22:24.185998 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:24.186969 (+ 971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:24.187343 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:24.187346 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:24.187348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:24.187423 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:24.187664 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:24.187667 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:24.187669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:24.187830 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:24.187914 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:24.188168 (+ 254us) server_negotiation.cc:299] Negotiation successful
0808 22:22:24.188266 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 22:22:24.192847 9923 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 22:22:24 dist-test-slave-ncp9 krb5kdc[21216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227744, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 22:22:24 dist-test-slave-ncp9 krb5kdc[21216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:22:24 dist-test-slave-ncp9 krb5kdc[21216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
W20260808 22:22:24.215467 21373 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:22:24.209998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58506 (local address 127.9.176.254:46215)
0808 22:22:24.210117 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:24.210120 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:24.210133 (+ 13us) server_negotiation.cc:407] Connection header received
0808 22:22:24.210172 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:24.210174 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:24.210209 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:24.210273 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:24.210903 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:24.211407 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:24.212159 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:24.212355 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:24.212418 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:24.215324 (+ 2906us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58506: BlockingRecv error: recv got EOF from 127.0.0.1:58506 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Aug 08 22:22:24 dist-test-slave-ncp9 krb5kdc[21216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227744, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM
I20260808 22:22:24.224169 21373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:24.216734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58514 (local address 127.9.176.254:46215)
0808 22:22:24.216856 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:24.216860 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:24.216977 (+ 117us) server_negotiation.cc:407] Connection header received
0808 22:22:24.217109 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:24.217112 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:24.217148 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:24.217219 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:24.217803 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:24.218296 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:24.219092 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:24.219311 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:24.219396 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:24.221686 (+ 2290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:24.221689 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:24.221691 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:22:24.221713 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:24.223019 (+ 1306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:24.223333 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:24.223335 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:24.223336 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:24.223401 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:24.223579 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:24.223581 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:24.223582 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:24.223731 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:24.223825 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:24.223964 (+ 139us) server_negotiation.cc:299] Negotiation successful
0808 22:22:24.224045 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 22:22:24.226869 21288 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58514:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 22:22:24.229858 21288 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 22:22:24.243364 22130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:24.237881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.194:36787 (local address 127.0.0.1:48144)
0808 22:22:24.238849 (+ 968us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:24.238862 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:24.238949 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:24.239626 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:24.239629 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:24.239644 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:22:24.239733 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:24.239740 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:24.240717 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:24.240720 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:24.241913 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:24.241923 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:24.242833 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:24.242836 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:24.242942 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:24.242984 (+ 42us) client_negotiation.cc:769] Sending connection context
0808 22:22:24.243096 (+ 112us) client_negotiation.cc:240] Negotiation successful
0808 22:22:24.243221 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":907,"thread_start_us":101,"threads_started":1}
I20260808 22:22:24.243395 22131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:24.238493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48144 (local address 127.9.176.194:36787)
0808 22:22:24.239208 (+ 715us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:24.239219 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:24.239240 (+ 21us) server_negotiation.cc:407] Connection header received
0808 22:22:24.239318 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:24.239323 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:24.239504 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:24.239631 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:22:24.239855 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:24.240605 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:24.242034 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:24.242709 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:24.242815 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:24.242976 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:24.243153 (+ 177us) server_negotiation.cc:299] Negotiation successful
0808 22:22:24.243250 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":101,"threads_started":1}
I20260808 22:22:24.243772 22128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:24.238447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.195:39019 (local address 127.0.0.1:41794)
0808 22:22:24.239246 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:24.239259 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:24.239350 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:24.239777 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:24.239780 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:24.239796 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:22:24.239869 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:24.239875 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:24.241164 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:24.241167 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:24.242340 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:24.242347 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:24.243432 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:24.243436 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:24.243522 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:24.243545 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 22:22:24.243606 (+ 61us) client_negotiation.cc:240] Negotiation successful
0808 22:22:24.243674 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":742,"mutex_wait_us":29,"thread_start_us":105,"threads_started":1}
I20260808 22:22:24.244114 22132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:24.238869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41794 (local address 127.9.176.195:39019)
0808 22:22:24.239226 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:24.239237 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:24.239327 (+ 90us) server_negotiation.cc:407] Connection header received
0808 22:22:24.239447 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:24.239452 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:24.239636 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:24.239776 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:22:24.239954 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:24.241044 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:24.242468 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:24.243124 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:24.243225 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:24.243368 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:24.243926 (+ 558us) server_negotiation.cc:299] Negotiation successful
0808 22:22:24.244011 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":119,"threads_started":1}
I20260808 22:22:24.246284 21743 tablet_service.cc:1467] Processing CreateTablet for tablet 7cae3298d14c46b49aa16a19d24f987f (DEFAULT_TABLE table=test-table [id=1f04acf99f214c2e841e170983724b86]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:22:24.246429 21968 tablet_service.cc:1467] Processing CreateTablet for tablet 7cae3298d14c46b49aa16a19d24f987f (DEFAULT_TABLE table=test-table [id=1f04acf99f214c2e841e170983724b86]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:22:24.247287 21968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cae3298d14c46b49aa16a19d24f987f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:22:24.247289 21743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cae3298d14c46b49aa16a19d24f987f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:22:24.249392 22126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:24.237746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.193:44137 (local address 127.0.0.1:45358)
0808 22:22:24.238474 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:24.238527 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:24.238665 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:24.239396 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:24.239402 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:24.239427 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:22:24.239558 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:24.239566 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:24.241035 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:24.241038 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:24.242328 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:24.242340 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:24.248988 (+ 6648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:24.248992 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:24.249096 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:24.249120 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 22:22:24.249194 (+ 74us) client_negotiation.cc:240] Negotiation successful
0808 22:22:24.249263 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"mutex_wait_us":37,"thread_start_us":62,"threads_started":1}
I20260808 22:22:24.249864 22127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:24.237881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45358 (local address 127.9.176.193:44137)
0808 22:22:24.238235 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:24.238252 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:24.238811 (+ 559us) server_negotiation.cc:407] Connection header received
0808 22:22:24.238894 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:24.238898 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:24.239071 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:24.239223 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:22:24.239704 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:24.240888 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:24.242461 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:24.243229 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:24.243328 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:24.249502 (+ 6174us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:24.249591 (+ 89us) server_negotiation.cc:299] Negotiation successful
0808 22:22:24.249737 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":62,"threads_started":1}
I20260808 22:22:24.252825 21528 tablet_service.cc:1467] Processing CreateTablet for tablet 7cae3298d14c46b49aa16a19d24f987f (DEFAULT_TABLE table=test-table [id=1f04acf99f214c2e841e170983724b86]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:22:24.253878 21528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cae3298d14c46b49aa16a19d24f987f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:22:24.254334 22135 tablet_bootstrap.cc:492] T 7cae3298d14c46b49aa16a19d24f987f P 3e22dfd749ff4f02b4c9ac55e651447e: Bootstrap starting.
I20260808 22:22:24.255549 22136 tablet_bootstrap.cc:492] T 7cae3298d14c46b49aa16a19d24f987f P 522121848ad548a49cf2f2ee25898ec3: Bootstrap starting.
I20260808 22:22:24.256598 22135 tablet_bootstrap.cc:654] T 7cae3298d14c46b49aa16a19d24f987f P 3e22dfd749ff4f02b4c9ac55e651447e: Neither blocks nor log segments found. Creating new log.
I20260808 22:22:24.257481 22135 log.cc:824] T 7cae3298d14c46b49aa16a19d24f987f P 3e22dfd749ff4f02b4c9ac55e651447e: Log is configured to *not* fsync() on all Append() calls
I20260808 22:22:24.257551 22136 tablet_bootstrap.cc:654] T 7cae3298d14c46b49aa16a19d24f987f P 522121848ad548a49cf2f2ee25898ec3: Neither blocks nor log segments found. Creating new log.
I20260808 22:22:24.258270 22136 log.cc:824] T 7cae3298d14c46b49aa16a19d24f987f P 522121848ad548a49cf2f2ee25898ec3: Log is configured to *not* fsync() on all Append() calls
I20260808 22:22:24.259724 22135 tablet_bootstrap.cc:492] T 7cae3298d14c46b49aa16a19d24f987f P 3e22dfd749ff4f02b4c9ac55e651447e: No bootstrap required, opened a new log
I20260808 22:22:24.259743 22140 tablet_bootstrap.cc:492] T 7cae3298d14c46b49aa16a19d24f987f P 7b49d3ab4f0c4429907d17e534a82add: Bootstrap starting.
I20260808 22:22:24.259915 22135 ts_tablet_manager.cc:1397] T 7cae3298d14c46b49aa16a19d24f987f P 3e22dfd749ff4f02b4c9ac55e651447e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 22:22:24.260186 22136 tablet_bootstrap.cc:492] T 7cae3298d14c46b49aa16a19d24f987f P 522121848ad548a49cf2f2ee25898ec3: No bootstrap required, opened a new log
I20260808 22:22:24.260372 22136 ts_tablet_manager.cc:1397] T 7cae3298d14c46b49aa16a19d24f987f P 522121848ad548a49cf2f2ee25898ec3: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260808 22:22:24.262069 22140 tablet_bootstrap.cc:654] T 7cae3298d14c46b49aa16a19d24f987f P 7b49d3ab4f0c4429907d17e534a82add: Neither blocks nor log segments found. Creating new log.
I20260808 22:22:24.262856 22135 raft_consensus.cc:348] T 7cae3298d14c46b49aa16a19d24f987f P 3e22dfd749ff4f02b4c9ac55e651447e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b49d3ab4f0c4429907d17e534a82add" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44137 } } peers { permanent_uuid: "3e22dfd749ff4f02b4c9ac55e651447e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36787 } } peers { permanent_uuid: "522121848ad548a49cf2f2ee25898ec3" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 39019 } }
I20260808 22:22:24.263057 22135 raft_consensus.cc:374] T 7cae3298d14c46b49aa16a19d24f987f P 3e22dfd749ff4f02b4c9ac55e651447e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:22:24.263109 22135 raft_consensus.cc:729] T 7cae3298d14c46b49aa16a19d24f987f P 3e22dfd749ff4f02b4c9ac55e651447e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e22dfd749ff4f02b4c9ac55e651447e, State: Initialized, Role: FOLLOWER
I20260808 22:22:24.263423 22140 log.cc:824] T 7cae3298d14c46b49aa16a19d24f987f P 7b49d3ab4f0c4429907d17e534a82add: Log is configured to *not* fsync() on all Append() calls
I20260808 22:22:24.263485 22135 consensus_queue.cc:260] T 7cae3298d14c46b49aa16a19d24f987f P 3e22dfd749ff4f02b4c9ac55e651447e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b49d3ab4f0c4429907d17e534a82add" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44137 } } peers { permanent_uuid: "3e22dfd749ff4f02b4c9ac55e651447e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36787 } } peers { permanent_uuid: "522121848ad548a49cf2f2ee25898ec3" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 39019 } }
I20260808 22:22:24.264066 22136 raft_consensus.cc:348] T 7cae3298d14c46b49aa16a19d24f987f P 522121848ad548a49cf2f2ee25898ec3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b49d3ab4f0c4429907d17e534a82add" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44137 } } peers { permanent_uuid: "3e22dfd749ff4f02b4c9ac55e651447e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36787 } } peers { permanent_uuid: "522121848ad548a49cf2f2ee25898ec3" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 39019 } }
I20260808 22:22:24.264315 22136 raft_consensus.cc:374] T 7cae3298d14c46b49aa16a19d24f987f P 522121848ad548a49cf2f2ee25898ec3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:22:24.264371 22136 raft_consensus.cc:729] T 7cae3298d14c46b49aa16a19d24f987f P 522121848ad548a49cf2f2ee25898ec3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522121848ad548a49cf2f2ee25898ec3, State: Initialized, Role: FOLLOWER
I20260808 22:22:24.264420 22135 ts_tablet_manager.cc:1428] T 7cae3298d14c46b49aa16a19d24f987f P 3e22dfd749ff4f02b4c9ac55e651447e: Time spent starting tablet: real 0.004s user 0.001s sys 0.005s
I20260808 22:22:24.264753 21852 heartbeater.cc:503] Master 127.9.176.254:46215 was elected leader, sending a full tablet report...
I20260808 22:22:24.264757 22136 consensus_queue.cc:260] T 7cae3298d14c46b49aa16a19d24f987f P 522121848ad548a49cf2f2ee25898ec3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b49d3ab4f0c4429907d17e534a82add" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44137 } } peers { permanent_uuid: "3e22dfd749ff4f02b4c9ac55e651447e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36787 } } peers { permanent_uuid: "522121848ad548a49cf2f2ee25898ec3" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 39019 } }
I20260808 22:22:24.265306 22140 tablet_bootstrap.cc:492] T 7cae3298d14c46b49aa16a19d24f987f P 7b49d3ab4f0c4429907d17e534a82add: No bootstrap required, opened a new log
I20260808 22:22:24.267000 22077 heartbeater.cc:503] Master 127.9.176.254:46215 was elected leader, sending a full tablet report...
I20260808 22:22:24.267580 22140 ts_tablet_manager.cc:1397] T 7cae3298d14c46b49aa16a19d24f987f P 7b49d3ab4f0c4429907d17e534a82add: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260808 22:22:24.267318 22136 ts_tablet_manager.cc:1428] T 7cae3298d14c46b49aa16a19d24f987f P 522121848ad548a49cf2f2ee25898ec3: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260808 22:22:24.270749 22140 raft_consensus.cc:348] T 7cae3298d14c46b49aa16a19d24f987f P 7b49d3ab4f0c4429907d17e534a82add [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b49d3ab4f0c4429907d17e534a82add" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44137 } } peers { permanent_uuid: "3e22dfd749ff4f02b4c9ac55e651447e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36787 } } peers { permanent_uuid: "522121848ad548a49cf2f2ee25898ec3" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 39019 } }
I20260808 22:22:24.270936 22140 raft_consensus.cc:374] T 7cae3298d14c46b49aa16a19d24f987f P 7b49d3ab4f0c4429907d17e534a82add [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:22:24.270970 22140 raft_consensus.cc:729] T 7cae3298d14c46b49aa16a19d24f987f P 7b49d3ab4f0c4429907d17e534a82add [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b49d3ab4f0c4429907d17e534a82add, State: Initialized, Role: FOLLOWER
I20260808 22:22:24.271301 22140 consensus_queue.cc:260] T 7cae3298d14c46b49aa16a19d24f987f P 7b49d3ab4f0c4429907d17e534a82add [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b49d3ab4f0c4429907d17e534a82add" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44137 } } peers { permanent_uuid: "3e22dfd749ff4f02b4c9ac55e651447e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36787 } } peers { permanent_uuid: "522121848ad548a49cf2f2ee25898ec3" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 39019 } }
I20260808 22:22:24.271839 21621 heartbeater.cc:503] Master 127.9.176.254:46215 was elected leader, sending a full tablet report...
I20260808 22:22:24.272075 22140 ts_tablet_manager.cc:1428] T 7cae3298d14c46b49aa16a19d24f987f P 7b49d3ab4f0c4429907d17e534a82add: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260808 22:22:24.313548 22145 raft_consensus.cc:482] T 7cae3298d14c46b49aa16a19d24f987f P 7b49d3ab4f0c4429907d17e534a82add [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 22:22:24.313721 22145 raft_consensus.cc:504] T 7cae3298d14c46b49aa16a19d24f987f P 7b49d3ab4f0c4429907d17e534a82add [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b49d3ab4f0c4429907d17e534a82add" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44137 } } peers { permanent_uuid: "3e22dfd749ff4f02b4c9ac55e651447e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36787 } } peers { permanent_uuid: "522121848ad548a49cf2f2ee25898ec3" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 39019 } }
I20260808 22:22:24.314797 22145 leader_election.cc:288] T 7cae3298d14c46b49aa16a19d24f987f P 7b49d3ab4f0c4429907d17e534a82add [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e22dfd749ff4f02b4c9ac55e651447e (127.9.176.194:36787), 522121848ad548a49cf2f2ee25898ec3 (127.9.176.195:39019)
I20260808 22:22:24.318635 22132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:24.315128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.193:38147 (local address 127.9.176.195:39019)
0808 22:22:24.315305 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:24.315309 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:24.315531 (+ 222us) server_negotiation.cc:407] Connection header received
0808 22:22:24.315629 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:24.315631 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:24.315692 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:24.315763 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:22:24.316030 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:24.316573 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:24.317733 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:24.318101 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:24.318196 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:24.318271 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:24.318441 (+ 170us) server_negotiation.cc:299] Negotiation successful
0808 22:22:24.318508 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260808 22:22:24.318661 22151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:24.314967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.194:36787 (local address 127.9.176.193:49641)
0808 22:22:24.315319 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:24.315332 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:24.315437 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:24.315679 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:24.315682 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:24.315697 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:22:24.315857 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:24.315866 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:24.316666 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:24.316669 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:24.317611 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:24.317621 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:24.318245 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:24.318249 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:24.318345 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:24.318364 (+ 19us) client_negotiation.cc:769] Sending connection context
0808 22:22:24.318432 (+ 68us) client_negotiation.cc:240] Negotiation successful
0808 22:22:24.318502 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"mutex_wait_us":42,"thread_start_us":116,"threads_started":1}
I20260808 22:22:24.318661 22131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:24.315116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.193:49641 (local address 127.9.176.194:36787)
0808 22:22:24.315295 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:24.315301 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:24.315393 (+ 92us) server_negotiation.cc:407] Connection header received
0808 22:22:24.315535 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:24.315538 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:24.315593 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:24.315658 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:22:24.315963 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:24.316563 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:24.317733 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:24.318130 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:24.318242 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:24.318293 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:24.318450 (+ 157us) server_negotiation.cc:299] Negotiation successful
0808 22:22:24.318540 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260808 22:22:24.318678 22148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:24.315037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.195:39019 (local address 127.9.176.193:38147)
0808 22:22:24.315440 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:24.315452 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:24.315548 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:24.315784 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:24.315787 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:24.315802 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:22:24.315926 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:24.315933 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:24.316679 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:24.316682 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:24.317617 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:24.317627 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:24.318209 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:24.318212 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:24.318302 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:24.318326 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 22:22:24.318413 (+ 87us) client_negotiation.cc:240] Negotiation successful
0808 22:22:24.318482 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"mutex_wait_us":61,"thread_start_us":106,"threads_started":1}
I20260808 22:22:24.319383 21772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cae3298d14c46b49aa16a19d24f987f" candidate_uuid: "7b49d3ab4f0c4429907d17e534a82add" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e22dfd749ff4f02b4c9ac55e651447e" is_pre_election: true
I20260808 22:22:24.319383 22000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cae3298d14c46b49aa16a19d24f987f" candidate_uuid: "7b49d3ab4f0c4429907d17e534a82add" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522121848ad548a49cf2f2ee25898ec3" is_pre_election: true
I20260808 22:22:24.319689 21772 raft_consensus.cc:2454] T 7cae3298d14c46b49aa16a19d24f987f P 3e22dfd749ff4f02b4c9ac55e651447e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b49d3ab4f0c4429907d17e534a82add in term 0.
I20260808 22:22:24.319689 22000 raft_consensus.cc:2454] T 7cae3298d14c46b49aa16a19d24f987f P 522121848ad548a49cf2f2ee25898ec3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b49d3ab4f0c4429907d17e534a82add in term 0.
I20260808 22:22:24.320175 21450 leader_election.cc:302] T 7cae3298d14c46b49aa16a19d24f987f P 7b49d3ab4f0c4429907d17e534a82add [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e22dfd749ff4f02b4c9ac55e651447e, 7b49d3ab4f0c4429907d17e534a82add; no voters:
I20260808 22:22:24.320401 22145 raft_consensus.cc:2781] T 7cae3298d14c46b49aa16a19d24f987f P 7b49d3ab4f0c4429907d17e534a82add [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 22:22:24.320478 22145 raft_consensus.cc:482] T 7cae3298d14c46b49aa16a19d24f987f P 7b49d3ab4f0c4429907d17e534a82add [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 22:22:24.320525 22145 raft_consensus.cc:3037] T 7cae3298d14c46b49aa16a19d24f987f P 7b49d3ab4f0c4429907d17e534a82add [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:22:24.321537 22145 raft_consensus.cc:504] T 7cae3298d14c46b49aa16a19d24f987f P 7b49d3ab4f0c4429907d17e534a82add [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b49d3ab4f0c4429907d17e534a82add" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44137 } } peers { permanent_uuid: "3e22dfd749ff4f02b4c9ac55e651447e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36787 } } peers { permanent_uuid: "522121848ad548a49cf2f2ee25898ec3" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 39019 } }
I20260808 22:22:24.321853 22145 leader_election.cc:288] T 7cae3298d14c46b49aa16a19d24f987f P 7b49d3ab4f0c4429907d17e534a82add [CANDIDATE]: Term 1 election: Requested vote from peers 3e22dfd749ff4f02b4c9ac55e651447e (127.9.176.194:36787), 522121848ad548a49cf2f2ee25898ec3 (127.9.176.195:39019)
I20260808 22:22:24.322153 22000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cae3298d14c46b49aa16a19d24f987f" candidate_uuid: "7b49d3ab4f0c4429907d17e534a82add" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522121848ad548a49cf2f2ee25898ec3"
I20260808 22:22:24.322176 21772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cae3298d14c46b49aa16a19d24f987f" candidate_uuid: "7b49d3ab4f0c4429907d17e534a82add" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e22dfd749ff4f02b4c9ac55e651447e"
I20260808 22:22:24.322274 22000 raft_consensus.cc:3037] T 7cae3298d14c46b49aa16a19d24f987f P 522121848ad548a49cf2f2ee25898ec3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:22:24.322291 21772 raft_consensus.cc:3037] T 7cae3298d14c46b49aa16a19d24f987f P 3e22dfd749ff4f02b4c9ac55e651447e [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:22:24.323267 22000 raft_consensus.cc:2454] T 7cae3298d14c46b49aa16a19d24f987f P 522121848ad548a49cf2f2ee25898ec3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b49d3ab4f0c4429907d17e534a82add in term 1.
I20260808 22:22:24.323565 21451 leader_election.cc:302] T 7cae3298d14c46b49aa16a19d24f987f P 7b49d3ab4f0c4429907d17e534a82add [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 522121848ad548a49cf2f2ee25898ec3, 7b49d3ab4f0c4429907d17e534a82add; no voters:
I20260808 22:22:24.323616 21772 raft_consensus.cc:2454] T 7cae3298d14c46b49aa16a19d24f987f P 3e22dfd749ff4f02b4c9ac55e651447e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b49d3ab4f0c4429907d17e534a82add in term 1.
I20260808 22:22:24.323724 22145 raft_consensus.cc:2781] T 7cae3298d14c46b49aa16a19d24f987f P 7b49d3ab4f0c4429907d17e534a82add [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:22:24.323910 22145 raft_consensus.cc:686] T 7cae3298d14c46b49aa16a19d24f987f P 7b49d3ab4f0c4429907d17e534a82add [term 1 LEADER]: Becoming Leader. State: Replica: 7b49d3ab4f0c4429907d17e534a82add, State: Running, Role: LEADER
I20260808 22:22:24.324213 22145 consensus_queue.cc:237] T 7cae3298d14c46b49aa16a19d24f987f P 7b49d3ab4f0c4429907d17e534a82add [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b49d3ab4f0c4429907d17e534a82add" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44137 } } peers { permanent_uuid: "3e22dfd749ff4f02b4c9ac55e651447e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36787 } } peers { permanent_uuid: "522121848ad548a49cf2f2ee25898ec3" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 39019 } }
I20260808 22:22:24.327198 21288 catalog_manager.cc:5495] T 7cae3298d14c46b49aa16a19d24f987f P 7b49d3ab4f0c4429907d17e534a82add reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b49d3ab4f0c4429907d17e534a82add (127.9.176.193). New cstate: current_term: 1 leader_uuid: "7b49d3ab4f0c4429907d17e534a82add" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b49d3ab4f0c4429907d17e534a82add" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44137 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e22dfd749ff4f02b4c9ac55e651447e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "522121848ad548a49cf2f2ee25898ec3" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 39019 } health_report { overall_health: UNKNOWN } } }
I20260808 22:22:24.338239 22127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:24.335083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45360 (local address 127.9.176.193:44137)
0808 22:22:24.335199 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:24.335202 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:24.335216 (+ 14us) server_negotiation.cc:407] Connection header received
0808 22:22:24.335353 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:24.335355 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:24.335401 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:24.335482 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:22:24.335689 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:24.336175 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:24.337028 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:24.337259 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:24.337333 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:24.337539 (+ 206us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:22:24.337936 (+ 397us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:22:24.338016 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:24.338088 (+ 72us) server_negotiation.cc:299] Negotiation successful
0808 22:22:24.338132 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 22:22:24.343623 21772 raft_consensus.cc:1261] T 7cae3298d14c46b49aa16a19d24f987f P 3e22dfd749ff4f02b4c9ac55e651447e [term 1 FOLLOWER]: Refusing update from remote peer 7b49d3ab4f0c4429907d17e534a82add: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:22:24.344038 22000 raft_consensus.cc:1261] T 7cae3298d14c46b49aa16a19d24f987f P 522121848ad548a49cf2f2ee25898ec3 [term 1 FOLLOWER]: Refusing update from remote peer 7b49d3ab4f0c4429907d17e534a82add: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:22:24.344285 22153 consensus_queue.cc:1035] T 7cae3298d14c46b49aa16a19d24f987f P 7b49d3ab4f0c4429907d17e534a82add [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e22dfd749ff4f02b4c9ac55e651447e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:22:24.344604 22145 consensus_queue.cc:1035] T 7cae3298d14c46b49aa16a19d24f987f P 7b49d3ab4f0c4429907d17e534a82add [LEADER]: Connected to new peer: Peer: permanent_uuid: "522121848ad548a49cf2f2ee25898ec3" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 39019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:22:24.350981 22157 mvcc.cc:205] Tried to move back new op lower bound from 7316388840825155584 to 7316388840754438144. Current Snapshot: MvccSnapshot[applied={T|T < 7316388840825155584}]
I20260808 22:22:24.351156 22158 mvcc.cc:205] Tried to move back new op lower bound from 7316388840825155584 to 7316388840754438144. Current Snapshot: MvccSnapshot[applied={T|T < 7316388840825155584}]
I20260808 22:22:24.353646 22159 mvcc.cc:205] Tried to move back new op lower bound from 7316388840825155584 to 7316388840754438144. Current Snapshot: MvccSnapshot[applied={T|T < 7316388840825155584}]
I20260808 22:22:24.358506 21287 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58514:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:22:24.358681 21287 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58514:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:22:24.360944 21287 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 824baa547e754adfbe38c0b466a74c12: Sending DeleteTablet for 3 replicas of tablet 7cae3298d14c46b49aa16a19d24f987f
I20260808 22:22:24.361661 21743 tablet_service.cc:1514] Processing DeleteTablet for tablet 7cae3298d14c46b49aa16a19d24f987f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:22:24 UTC) from {username='oryx', principal='oryx/127.9.176.254@KRBTEST.COM'} at 127.0.0.1:48144
I20260808 22:22:24.361701 21528 tablet_service.cc:1514] Processing DeleteTablet for tablet 7cae3298d14c46b49aa16a19d24f987f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:22:24 UTC) from {username='oryx', principal='oryx/127.9.176.254@KRBTEST.COM'} at 127.0.0.1:45358
I20260808 22:22:24.361740 21968 tablet_service.cc:1514] Processing DeleteTablet for tablet 7cae3298d14c46b49aa16a19d24f987f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:22:24 UTC) from {username='oryx', principal='oryx/127.9.176.254@KRBTEST.COM'} at 127.0.0.1:41794
I20260808 22:22:24.362115 22170 tablet_replica.cc:330] T 7cae3298d14c46b49aa16a19d24f987f P 3e22dfd749ff4f02b4c9ac55e651447e: stopping tablet replica
I20260808 22:22:24.362345 22172 tablet_replica.cc:330] T 7cae3298d14c46b49aa16a19d24f987f P 7b49d3ab4f0c4429907d17e534a82add: stopping tablet replica
I20260808 22:22:24.362413 22170 raft_consensus.cc:2229] T 7cae3298d14c46b49aa16a19d24f987f P 3e22dfd749ff4f02b4c9ac55e651447e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:22:24.362572 22175 tablet_replica.cc:330] T 7cae3298d14c46b49aa16a19d24f987f P 522121848ad548a49cf2f2ee25898ec3: stopping tablet replica
I20260808 22:22:24.362710 22170 raft_consensus.cc:2258] T 7cae3298d14c46b49aa16a19d24f987f P 3e22dfd749ff4f02b4c9ac55e651447e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:22:24.362871 21287 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58514:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 22:22:24.363267 21287 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 22:22:24.363282 22175 raft_consensus.cc:2229] T 7cae3298d14c46b49aa16a19d24f987f P 522121848ad548a49cf2f2ee25898ec3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:22:24.363548 22172 raft_consensus.cc:2229] T 7cae3298d14c46b49aa16a19d24f987f P 7b49d3ab4f0c4429907d17e534a82add [term 1 LEADER]: Raft consensus shutting down.
I20260808 22:22:24.363895 22175 raft_consensus.cc:2258] T 7cae3298d14c46b49aa16a19d24f987f P 522121848ad548a49cf2f2ee25898ec3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:22:24.364168 22170 ts_tablet_manager.cc:1905] T 7cae3298d14c46b49aa16a19d24f987f P 3e22dfd749ff4f02b4c9ac55e651447e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:22:24.364300 22172 raft_consensus.cc:2258] T 7cae3298d14c46b49aa16a19d24f987f P 7b49d3ab4f0c4429907d17e534a82add [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:22:24.365913 22175 ts_tablet_manager.cc:1905] T 7cae3298d14c46b49aa16a19d24f987f P 522121848ad548a49cf2f2ee25898ec3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:22:24.366900 22170 ts_tablet_manager.cc:1918] T 7cae3298d14c46b49aa16a19d24f987f P 3e22dfd749ff4f02b4c9ac55e651447e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 22:22:24.367000 22170 log.cc:1196] T 7cae3298d14c46b49aa16a19d24f987f P 3e22dfd749ff4f02b4c9ac55e651447e: Deleting WAL directory at /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/minicluster-data/ts-1/wal/wals/7cae3298d14c46b49aa16a19d24f987f
I20260808 22:22:24.367316 22170 ts_tablet_manager.cc:1939] T 7cae3298d14c46b49aa16a19d24f987f P 3e22dfd749ff4f02b4c9ac55e651447e: Deleting consensus metadata
I20260808 22:22:24.368640 22175 ts_tablet_manager.cc:1918] T 7cae3298d14c46b49aa16a19d24f987f P 522121848ad548a49cf2f2ee25898ec3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 22:22:24.368701 21743 tablet_service.cc:1467] Processing CreateTablet for tablet 2a7e860cdb784ca4bee6e2744670d977 (DEFAULT_TABLE table=test-table [id=459b6357f5654d77aea834d11e1cd95f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:22:24.368852 22175 log.cc:1196] T 7cae3298d14c46b49aa16a19d24f987f P 522121848ad548a49cf2f2ee25898ec3: Deleting WAL directory at /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/minicluster-data/ts-2/wal/wals/7cae3298d14c46b49aa16a19d24f987f
I20260808 22:22:24.368986 21743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a7e860cdb784ca4bee6e2744670d977. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:22:24.368988 21256 catalog_manager.cc:4841] TS 3e22dfd749ff4f02b4c9ac55e651447e (127.9.176.194:36787): tablet 7cae3298d14c46b49aa16a19d24f987f (table test-table [id=1f04acf99f214c2e841e170983724b86]) successfully deleted
I20260808 22:22:24.369215 21968 tablet_service.cc:1467] Processing CreateTablet for tablet 2a7e860cdb784ca4bee6e2744670d977 (DEFAULT_TABLE table=test-table [id=459b6357f5654d77aea834d11e1cd95f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:22:24.369446 21968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a7e860cdb784ca4bee6e2744670d977. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:22:24.369959 21528 tablet_service.cc:1467] Processing CreateTablet for tablet 2a7e860cdb784ca4bee6e2744670d977 (DEFAULT_TABLE table=test-table [id=459b6357f5654d77aea834d11e1cd95f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:22:24.370210 21528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a7e860cdb784ca4bee6e2744670d977. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:22:24.371445 22136 tablet_bootstrap.cc:492] T 2a7e860cdb784ca4bee6e2744670d977 P 522121848ad548a49cf2f2ee25898ec3: Bootstrap starting.
I20260808 22:22:24.371537 22135 tablet_bootstrap.cc:492] T 2a7e860cdb784ca4bee6e2744670d977 P 3e22dfd749ff4f02b4c9ac55e651447e: Bootstrap starting.
I20260808 22:22:24.372373 22136 tablet_bootstrap.cc:654] T 2a7e860cdb784ca4bee6e2744670d977 P 522121848ad548a49cf2f2ee25898ec3: Neither blocks nor log segments found. Creating new log.
I20260808 22:22:24.372444 22135 tablet_bootstrap.cc:654] T 2a7e860cdb784ca4bee6e2744670d977 P 3e22dfd749ff4f02b4c9ac55e651447e: Neither blocks nor log segments found. Creating new log.
I20260808 22:22:24.372479 22140 tablet_bootstrap.cc:492] T 2a7e860cdb784ca4bee6e2744670d977 P 7b49d3ab4f0c4429907d17e534a82add: Bootstrap starting.
I20260808 22:22:24.369448 22175 ts_tablet_manager.cc:1939] T 7cae3298d14c46b49aa16a19d24f987f P 522121848ad548a49cf2f2ee25898ec3: Deleting consensus metadata
I20260808 22:22:24.373365 22140 tablet_bootstrap.cc:654] T 2a7e860cdb784ca4bee6e2744670d977 P 7b49d3ab4f0c4429907d17e534a82add: Neither blocks nor log segments found. Creating new log.
I20260808 22:22:24.373498 22136 tablet_bootstrap.cc:492] T 2a7e860cdb784ca4bee6e2744670d977 P 522121848ad548a49cf2f2ee25898ec3: No bootstrap required, opened a new log
I20260808 22:22:24.373597 22136 ts_tablet_manager.cc:1397] T 2a7e860cdb784ca4bee6e2744670d977 P 522121848ad548a49cf2f2ee25898ec3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 22:22:24.374030 22136 raft_consensus.cc:348] T 2a7e860cdb784ca4bee6e2744670d977 P 522121848ad548a49cf2f2ee25898ec3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b49d3ab4f0c4429907d17e534a82add" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44137 } } peers { permanent_uuid: "3e22dfd749ff4f02b4c9ac55e651447e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36787 } } peers { permanent_uuid: "522121848ad548a49cf2f2ee25898ec3" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 39019 } }
I20260808 22:22:24.374140 22136 raft_consensus.cc:374] T 2a7e860cdb784ca4bee6e2744670d977 P 522121848ad548a49cf2f2ee25898ec3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:22:24.374174 22136 raft_consensus.cc:729] T 2a7e860cdb784ca4bee6e2744670d977 P 522121848ad548a49cf2f2ee25898ec3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522121848ad548a49cf2f2ee25898ec3, State: Initialized, Role: FOLLOWER
I20260808 22:22:24.374240 22172 ts_tablet_manager.cc:1905] T 7cae3298d14c46b49aa16a19d24f987f P 7b49d3ab4f0c4429907d17e534a82add: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:22:24.374284 22136 consensus_queue.cc:260] T 2a7e860cdb784ca4bee6e2744670d977 P 522121848ad548a49cf2f2ee25898ec3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b49d3ab4f0c4429907d17e534a82add" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44137 } } peers { permanent_uuid: "3e22dfd749ff4f02b4c9ac55e651447e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36787 } } peers { permanent_uuid: "522121848ad548a49cf2f2ee25898ec3" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 39019 } }
I20260808 22:22:24.374517 22136 ts_tablet_manager.cc:1428] T 2a7e860cdb784ca4bee6e2744670d977 P 522121848ad548a49cf2f2ee25898ec3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 22:22:24.375144 21257 catalog_manager.cc:4841] TS 522121848ad548a49cf2f2ee25898ec3 (127.9.176.195:39019): tablet 7cae3298d14c46b49aa16a19d24f987f (table test-table [id=1f04acf99f214c2e841e170983724b86]) successfully deleted
I20260808 22:22:24.375747 22135 tablet_bootstrap.cc:492] T 2a7e860cdb784ca4bee6e2744670d977 P 3e22dfd749ff4f02b4c9ac55e651447e: No bootstrap required, opened a new log
I20260808 22:22:24.375828 22135 ts_tablet_manager.cc:1397] T 2a7e860cdb784ca4bee6e2744670d977 P 3e22dfd749ff4f02b4c9ac55e651447e: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260808 22:22:24.376262 22135 raft_consensus.cc:348] T 2a7e860cdb784ca4bee6e2744670d977 P 3e22dfd749ff4f02b4c9ac55e651447e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b49d3ab4f0c4429907d17e534a82add" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44137 } } peers { permanent_uuid: "3e22dfd749ff4f02b4c9ac55e651447e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36787 } } peers { permanent_uuid: "522121848ad548a49cf2f2ee25898ec3" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 39019 } }
I20260808 22:22:24.376362 22135 raft_consensus.cc:374] T 2a7e860cdb784ca4bee6e2744670d977 P 3e22dfd749ff4f02b4c9ac55e651447e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:22:24.376394 22135 raft_consensus.cc:729] T 2a7e860cdb784ca4bee6e2744670d977 P 3e22dfd749ff4f02b4c9ac55e651447e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e22dfd749ff4f02b4c9ac55e651447e, State: Initialized, Role: FOLLOWER
I20260808 22:22:24.376510 22135 consensus_queue.cc:260] T 2a7e860cdb784ca4bee6e2744670d977 P 3e22dfd749ff4f02b4c9ac55e651447e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b49d3ab4f0c4429907d17e534a82add" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44137 } } peers { permanent_uuid: "3e22dfd749ff4f02b4c9ac55e651447e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36787 } } peers { permanent_uuid: "522121848ad548a49cf2f2ee25898ec3" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 39019 } }
I20260808 22:22:24.376760 22135 ts_tablet_manager.cc:1428] T 2a7e860cdb784ca4bee6e2744670d977 P 3e22dfd749ff4f02b4c9ac55e651447e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260808 22:22:24.377736 22140 tablet_bootstrap.cc:492] T 2a7e860cdb784ca4bee6e2744670d977 P 7b49d3ab4f0c4429907d17e534a82add: No bootstrap required, opened a new log
I20260808 22:22:24.377827 22140 ts_tablet_manager.cc:1397] T 2a7e860cdb784ca4bee6e2744670d977 P 7b49d3ab4f0c4429907d17e534a82add: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260808 22:22:24.377980 22172 ts_tablet_manager.cc:1918] T 7cae3298d14c46b49aa16a19d24f987f P 7b49d3ab4f0c4429907d17e534a82add: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 22:22:24.378077 22172 log.cc:1196] T 7cae3298d14c46b49aa16a19d24f987f P 7b49d3ab4f0c4429907d17e534a82add: Deleting WAL directory at /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/minicluster-data/ts-0/wal/wals/7cae3298d14c46b49aa16a19d24f987f
I20260808 22:22:24.378261 22140 raft_consensus.cc:348] T 2a7e860cdb784ca4bee6e2744670d977 P 7b49d3ab4f0c4429907d17e534a82add [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b49d3ab4f0c4429907d17e534a82add" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44137 } } peers { permanent_uuid: "3e22dfd749ff4f02b4c9ac55e651447e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36787 } } peers { permanent_uuid: "522121848ad548a49cf2f2ee25898ec3" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 39019 } }
I20260808 22:22:24.378330 22172 ts_tablet_manager.cc:1939] T 7cae3298d14c46b49aa16a19d24f987f P 7b49d3ab4f0c4429907d17e534a82add: Deleting consensus metadata
I20260808 22:22:24.378377 22140 raft_consensus.cc:374] T 2a7e860cdb784ca4bee6e2744670d977 P 7b49d3ab4f0c4429907d17e534a82add [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:22:24.378417 22140 raft_consensus.cc:729] T 2a7e860cdb784ca4bee6e2744670d977 P 7b49d3ab4f0c4429907d17e534a82add [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b49d3ab4f0c4429907d17e534a82add, State: Initialized, Role: FOLLOWER
I20260808 22:22:24.378525 22140 consensus_queue.cc:260] T 2a7e860cdb784ca4bee6e2744670d977 P 7b49d3ab4f0c4429907d17e534a82add [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b49d3ab4f0c4429907d17e534a82add" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44137 } } peers { permanent_uuid: "3e22dfd749ff4f02b4c9ac55e651447e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36787 } } peers { permanent_uuid: "522121848ad548a49cf2f2ee25898ec3" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 39019 } }
I20260808 22:22:24.378769 22140 ts_tablet_manager.cc:1428] T 2a7e860cdb784ca4bee6e2744670d977 P 7b49d3ab4f0c4429907d17e534a82add: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 22:22:24.379246 21258 catalog_manager.cc:4841] TS 7b49d3ab4f0c4429907d17e534a82add (127.9.176.193:44137): tablet 7cae3298d14c46b49aa16a19d24f987f (table test-table [id=1f04acf99f214c2e841e170983724b86]) successfully deleted
W20260808 22:22:24.427534 22081 tablet.cc:2369] T 2a7e860cdb784ca4bee6e2744670d977 P 522121848ad548a49cf2f2ee25898ec3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 22:22:24.472983 21854 tablet.cc:2369] T 2a7e860cdb784ca4bee6e2744670d977 P 3e22dfd749ff4f02b4c9ac55e651447e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 22:22:24.494947 21622 tablet.cc:2369] T 2a7e860cdb784ca4bee6e2744670d977 P 7b49d3ab4f0c4429907d17e534a82add: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 22:22:24.560276 22143 raft_consensus.cc:482] T 2a7e860cdb784ca4bee6e2744670d977 P 3e22dfd749ff4f02b4c9ac55e651447e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 22:22:24.560433 22143 raft_consensus.cc:504] T 2a7e860cdb784ca4bee6e2744670d977 P 3e22dfd749ff4f02b4c9ac55e651447e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b49d3ab4f0c4429907d17e534a82add" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44137 } } peers { permanent_uuid: "3e22dfd749ff4f02b4c9ac55e651447e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36787 } } peers { permanent_uuid: "522121848ad548a49cf2f2ee25898ec3" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 39019 } }
I20260808 22:22:24.561515 22143 leader_election.cc:288] T 2a7e860cdb784ca4bee6e2744670d977 P 3e22dfd749ff4f02b4c9ac55e651447e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b49d3ab4f0c4429907d17e534a82add (127.9.176.193:44137), 522121848ad548a49cf2f2ee25898ec3 (127.9.176.195:39019)
I20260808 22:22:24.565279 22127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:24.561862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.194:45303 (local address 127.9.176.193:44137)
0808 22:22:24.561984 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:24.561988 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:24.562164 (+ 176us) server_negotiation.cc:407] Connection header received
0808 22:22:24.562276 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:24.562280 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:24.562330 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:24.562398 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:22:24.562707 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:24.563214 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:24.564350 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:24.564749 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:24.564861 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:24.564939 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:24.565095 (+ 156us) server_negotiation.cc:299] Negotiation successful
0808 22:22:24.565166 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 22:22:24.565280 22132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:24.561843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.194:44265 (local address 127.9.176.195:39019)
0808 22:22:24.561985 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:24.561989 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:24.562260 (+ 271us) server_negotiation.cc:407] Connection header received
0808 22:22:24.562363 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:24.562365 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:24.562413 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:24.562488 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:22:24.562733 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:24.563221 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:24.564340 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:24.564719 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:24.564811 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:24.564865 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:24.565100 (+ 235us) server_negotiation.cc:299] Negotiation successful
0808 22:22:24.565165 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260808 22:22:24.565301 22180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:24.561728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.195:39019 (local address 127.9.176.194:44265)
0808 22:22:24.562131 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:24.562145 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:24.562276 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:24.562515 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:24.562517 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:24.562531 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:22:24.562627 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:24.562632 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:24.563333 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:24.563336 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:24.564217 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:24.564226 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:24.564837 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:24.564841 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:24.564969 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:24.564995 (+ 26us) client_negotiation.cc:769] Sending connection context
0808 22:22:24.565061 (+ 66us) client_negotiation.cc:240] Negotiation successful
0808 22:22:24.565135 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"mutex_wait_us":77,"thread_start_us":80,"threads_started":1}
I20260808 22:22:24.565308 22179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:24.561728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.193:44137 (local address 127.9.176.194:45303)
0808 22:22:24.562070 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:24.562082 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:24.562176 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:24.562416 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:24.562419 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:24.562435 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:22:24.562584 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:24.562590 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:24.563318 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:24.563322 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:24.564223 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:24.564232 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:24.564865 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:24.564869 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:24.564979 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:24.564998 (+ 19us) client_negotiation.cc:769] Sending connection context
0808 22:22:24.565067 (+ 69us) client_negotiation.cc:240] Negotiation successful
0808 22:22:24.565138 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"mutex_wait_us":57,"thread_start_us":86,"threads_started":1}
I20260808 22:22:24.565840 22000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a7e860cdb784ca4bee6e2744670d977" candidate_uuid: "3e22dfd749ff4f02b4c9ac55e651447e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522121848ad548a49cf2f2ee25898ec3" is_pre_election: true
I20260808 22:22:24.565996 22000 raft_consensus.cc:2454] T 2a7e860cdb784ca4bee6e2744670d977 P 522121848ad548a49cf2f2ee25898ec3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e22dfd749ff4f02b4c9ac55e651447e in term 0.
I20260808 22:22:24.565990 21558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a7e860cdb784ca4bee6e2744670d977" candidate_uuid: "3e22dfd749ff4f02b4c9ac55e651447e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b49d3ab4f0c4429907d17e534a82add" is_pre_election: true
I20260808 22:22:24.566217 21558 raft_consensus.cc:2454] T 2a7e860cdb784ca4bee6e2744670d977 P 7b49d3ab4f0c4429907d17e534a82add [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e22dfd749ff4f02b4c9ac55e651447e in term 0.
I20260808 22:22:24.566351 21660 leader_election.cc:302] T 2a7e860cdb784ca4bee6e2744670d977 P 3e22dfd749ff4f02b4c9ac55e651447e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e22dfd749ff4f02b4c9ac55e651447e, 522121848ad548a49cf2f2ee25898ec3; no voters:
I20260808 22:22:24.566596 22143 raft_consensus.cc:2781] T 2a7e860cdb784ca4bee6e2744670d977 P 3e22dfd749ff4f02b4c9ac55e651447e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 22:22:24.566663 22143 raft_consensus.cc:482] T 2a7e860cdb784ca4bee6e2744670d977 P 3e22dfd749ff4f02b4c9ac55e651447e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 22:22:24.566708 22143 raft_consensus.cc:3037] T 2a7e860cdb784ca4bee6e2744670d977 P 3e22dfd749ff4f02b4c9ac55e651447e [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:22:24.567672 22143 raft_consensus.cc:504] T 2a7e860cdb784ca4bee6e2744670d977 P 3e22dfd749ff4f02b4c9ac55e651447e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b49d3ab4f0c4429907d17e534a82add" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44137 } } peers { permanent_uuid: "3e22dfd749ff4f02b4c9ac55e651447e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36787 } } peers { permanent_uuid: "522121848ad548a49cf2f2ee25898ec3" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 39019 } }
I20260808 22:22:24.567962 22143 leader_election.cc:288] T 2a7e860cdb784ca4bee6e2744670d977 P 3e22dfd749ff4f02b4c9ac55e651447e [CANDIDATE]: Term 1 election: Requested vote from peers 7b49d3ab4f0c4429907d17e534a82add (127.9.176.193:44137), 522121848ad548a49cf2f2ee25898ec3 (127.9.176.195:39019)
I20260808 22:22:24.568279 21558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a7e860cdb784ca4bee6e2744670d977" candidate_uuid: "3e22dfd749ff4f02b4c9ac55e651447e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b49d3ab4f0c4429907d17e534a82add"
I20260808 22:22:24.568292 22000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a7e860cdb784ca4bee6e2744670d977" candidate_uuid: "3e22dfd749ff4f02b4c9ac55e651447e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522121848ad548a49cf2f2ee25898ec3"
I20260808 22:22:24.568395 22000 raft_consensus.cc:3037] T 2a7e860cdb784ca4bee6e2744670d977 P 522121848ad548a49cf2f2ee25898ec3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:22:24.568395 21558 raft_consensus.cc:3037] T 2a7e860cdb784ca4bee6e2744670d977 P 7b49d3ab4f0c4429907d17e534a82add [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:22:24.569530 21558 raft_consensus.cc:2454] T 2a7e860cdb784ca4bee6e2744670d977 P 7b49d3ab4f0c4429907d17e534a82add [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e22dfd749ff4f02b4c9ac55e651447e in term 1.
I20260808 22:22:24.569530 22000 raft_consensus.cc:2454] T 2a7e860cdb784ca4bee6e2744670d977 P 522121848ad548a49cf2f2ee25898ec3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e22dfd749ff4f02b4c9ac55e651447e in term 1.
I20260808 22:22:24.569806 21660 leader_election.cc:302] T 2a7e860cdb784ca4bee6e2744670d977 P 3e22dfd749ff4f02b4c9ac55e651447e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3e22dfd749ff4f02b4c9ac55e651447e, 522121848ad548a49cf2f2ee25898ec3, 7b49d3ab4f0c4429907d17e534a82add; no voters:
I20260808 22:22:24.569975 22143 raft_consensus.cc:2781] T 2a7e860cdb784ca4bee6e2744670d977 P 3e22dfd749ff4f02b4c9ac55e651447e [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:22:24.570190 22143 raft_consensus.cc:686] T 2a7e860cdb784ca4bee6e2744670d977 P 3e22dfd749ff4f02b4c9ac55e651447e [term 1 LEADER]: Becoming Leader. State: Replica: 3e22dfd749ff4f02b4c9ac55e651447e, State: Running, Role: LEADER
I20260808 22:22:24.570441 22143 consensus_queue.cc:237] T 2a7e860cdb784ca4bee6e2744670d977 P 3e22dfd749ff4f02b4c9ac55e651447e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b49d3ab4f0c4429907d17e534a82add" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44137 } } peers { permanent_uuid: "3e22dfd749ff4f02b4c9ac55e651447e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36787 } } peers { permanent_uuid: "522121848ad548a49cf2f2ee25898ec3" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 39019 } }
I20260808 22:22:24.572573 21288 catalog_manager.cc:5495] T 2a7e860cdb784ca4bee6e2744670d977 P 3e22dfd749ff4f02b4c9ac55e651447e reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e22dfd749ff4f02b4c9ac55e651447e (127.9.176.194). New cstate: current_term: 1 leader_uuid: "3e22dfd749ff4f02b4c9ac55e651447e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b49d3ab4f0c4429907d17e534a82add" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44137 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e22dfd749ff4f02b4c9ac55e651447e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36787 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "522121848ad548a49cf2f2ee25898ec3" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 39019 } health_report { overall_health: UNKNOWN } } }
Aug 08 22:22:24 dist-test-slave-ncp9 krb5kdc[21216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227743, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.176.254@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM
I20260808 22:22:24.634225 22190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:24.626529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:46215 (local address 127.0.0.1:58526)
0808 22:22:24.626826 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:24.626843 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:24.626955 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:24.627202 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:24.627205 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:24.627620 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:24.627729 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:24.627736 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:24.628579 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:24.628582 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:24.629188 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:24.629198 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:24.629606 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:24.629609 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:24.629694 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:24.630210 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:24.630222 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:24.631562 (+ 1340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:22:24.632922 (+ 1360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:24.632933 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:24.632941 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:24.633171 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:24.633394 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:24.633395 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:24.633397 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:24.633468 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:24.633758 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:22:24.633763 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:22:24.633864 (+ 101us) client_negotiation.cc:769] Sending connection context
0808 22:22:24.633949 (+ 85us) client_negotiation.cc:240] Negotiation successful
0808 22:22:24.634065 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":101,"threads_started":1}
I20260808 22:22:24.634245 21373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:24.626587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58526 (local address 127.9.176.254:46215)
0808 22:22:24.626726 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:24.626731 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:24.626915 (+ 184us) server_negotiation.cc:407] Connection header received
0808 22:22:24.627058 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:24.627061 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:24.627104 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:24.627181 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:24.627839 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:24.628479 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:24.629301 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:24.629516 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:24.629598 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:24.631704 (+ 2106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:24.631708 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:24.631710 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:22:24.631731 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:24.632802 (+ 1071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:24.633255 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:24.633258 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:24.633259 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:24.633330 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:24.633525 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:24.633526 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:24.633527 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:24.633686 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:24.633758 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:24.633988 (+ 230us) server_negotiation.cc:299] Negotiation successful
0808 22:22:24.634122 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":57}
I20260808 22:22:24.637969 21288 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.9.176.254@KRBTEST.COM'} at 127.0.0.1:58526:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260808 22:22:24.638666 21288 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 22:22:24.643188 21528 tablet_service.cc:1467] Processing CreateTablet for tablet ded13fb73b54433b8d177d369ef57231 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cfb3c2d9e4c74981b8bb2020231eab0a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 22:22:24.643208 21743 tablet_service.cc:1467] Processing CreateTablet for tablet ded13fb73b54433b8d177d369ef57231 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cfb3c2d9e4c74981b8bb2020231eab0a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 22:22:24.643222 21968 tablet_service.cc:1467] Processing CreateTablet for tablet ded13fb73b54433b8d177d369ef57231 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cfb3c2d9e4c74981b8bb2020231eab0a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 22:22:24.643596 21528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ded13fb73b54433b8d177d369ef57231. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:22:24.643596 21743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ded13fb73b54433b8d177d369ef57231. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:22:24.643615 21968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ded13fb73b54433b8d177d369ef57231. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:22:24.646255 22136 tablet_bootstrap.cc:492] T ded13fb73b54433b8d177d369ef57231 P 522121848ad548a49cf2f2ee25898ec3: Bootstrap starting.
I20260808 22:22:24.646327 22140 tablet_bootstrap.cc:492] T ded13fb73b54433b8d177d369ef57231 P 7b49d3ab4f0c4429907d17e534a82add: Bootstrap starting.
I20260808 22:22:24.647097 22140 tablet_bootstrap.cc:654] T ded13fb73b54433b8d177d369ef57231 P 7b49d3ab4f0c4429907d17e534a82add: Neither blocks nor log segments found. Creating new log.
I20260808 22:22:24.647158 22136 tablet_bootstrap.cc:654] T ded13fb73b54433b8d177d369ef57231 P 522121848ad548a49cf2f2ee25898ec3: Neither blocks nor log segments found. Creating new log.
I20260808 22:22:24.647732 22135 tablet_bootstrap.cc:492] T ded13fb73b54433b8d177d369ef57231 P 3e22dfd749ff4f02b4c9ac55e651447e: Bootstrap starting.
I20260808 22:22:24.648272 22140 tablet_bootstrap.cc:492] T ded13fb73b54433b8d177d369ef57231 P 7b49d3ab4f0c4429907d17e534a82add: No bootstrap required, opened a new log
I20260808 22:22:24.648356 22140 ts_tablet_manager.cc:1397] T ded13fb73b54433b8d177d369ef57231 P 7b49d3ab4f0c4429907d17e534a82add: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 22:22:24.648705 22135 tablet_bootstrap.cc:654] T ded13fb73b54433b8d177d369ef57231 P 3e22dfd749ff4f02b4c9ac55e651447e: Neither blocks nor log segments found. Creating new log.
I20260808 22:22:24.648794 22140 raft_consensus.cc:348] T ded13fb73b54433b8d177d369ef57231 P 7b49d3ab4f0c4429907d17e534a82add [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b49d3ab4f0c4429907d17e534a82add" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44137 } } peers { permanent_uuid: "3e22dfd749ff4f02b4c9ac55e651447e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36787 } } peers { permanent_uuid: "522121848ad548a49cf2f2ee25898ec3" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 39019 } }
I20260808 22:22:24.648932 22140 raft_consensus.cc:374] T ded13fb73b54433b8d177d369ef57231 P 7b49d3ab4f0c4429907d17e534a82add [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:22:24.648967 22140 raft_consensus.cc:729] T ded13fb73b54433b8d177d369ef57231 P 7b49d3ab4f0c4429907d17e534a82add [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b49d3ab4f0c4429907d17e534a82add, State: Initialized, Role: FOLLOWER
I20260808 22:22:24.649093 22140 consensus_queue.cc:260] T ded13fb73b54433b8d177d369ef57231 P 7b49d3ab4f0c4429907d17e534a82add [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b49d3ab4f0c4429907d17e534a82add" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44137 } } peers { permanent_uuid: "3e22dfd749ff4f02b4c9ac55e651447e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36787 } } peers { permanent_uuid: "522121848ad548a49cf2f2ee25898ec3" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 39019 } }
I20260808 22:22:24.649371 22140 ts_tablet_manager.cc:1428] T ded13fb73b54433b8d177d369ef57231 P 7b49d3ab4f0c4429907d17e534a82add: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 22:22:24.649384 22145 tablet_replica.cc:439] T ded13fb73b54433b8d177d369ef57231 P 7b49d3ab4f0c4429907d17e534a82add: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b49d3ab4f0c4429907d17e534a82add" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44137 } } peers { permanent_uuid: "3e22dfd749ff4f02b4c9ac55e651447e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36787 } } peers { permanent_uuid: "522121848ad548a49cf2f2ee25898ec3" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 39019 } } }
I20260808 22:22:24.649547 22145 tablet_replica.cc:442] T ded13fb73b54433b8d177d369ef57231 P 7b49d3ab4f0c4429907d17e534a82add: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 22:22:24.649698 22136 tablet_bootstrap.cc:492] T ded13fb73b54433b8d177d369ef57231 P 522121848ad548a49cf2f2ee25898ec3: No bootstrap required, opened a new log
I20260808 22:22:24.649770 22136 ts_tablet_manager.cc:1397] T ded13fb73b54433b8d177d369ef57231 P 522121848ad548a49cf2f2ee25898ec3: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260808 22:22:24.650174 22136 raft_consensus.cc:348] T ded13fb73b54433b8d177d369ef57231 P 522121848ad548a49cf2f2ee25898ec3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b49d3ab4f0c4429907d17e534a82add" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44137 } } peers { permanent_uuid: "3e22dfd749ff4f02b4c9ac55e651447e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36787 } } peers { permanent_uuid: "522121848ad548a49cf2f2ee25898ec3" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 39019 } }
I20260808 22:22:24.650285 22136 raft_consensus.cc:374] T ded13fb73b54433b8d177d369ef57231 P 522121848ad548a49cf2f2ee25898ec3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:22:24.650322 22136 raft_consensus.cc:729] T ded13fb73b54433b8d177d369ef57231 P 522121848ad548a49cf2f2ee25898ec3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522121848ad548a49cf2f2ee25898ec3, State: Initialized, Role: FOLLOWER
I20260808 22:22:24.650435 22136 consensus_queue.cc:260] T ded13fb73b54433b8d177d369ef57231 P 522121848ad548a49cf2f2ee25898ec3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b49d3ab4f0c4429907d17e534a82add" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44137 } } peers { permanent_uuid: "3e22dfd749ff4f02b4c9ac55e651447e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36787 } } peers { permanent_uuid: "522121848ad548a49cf2f2ee25898ec3" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 39019 } }
I20260808 22:22:24.650633 22135 tablet_bootstrap.cc:492] T ded13fb73b54433b8d177d369ef57231 P 3e22dfd749ff4f02b4c9ac55e651447e: No bootstrap required, opened a new log
I20260808 22:22:24.650668 22136 ts_tablet_manager.cc:1428] T ded13fb73b54433b8d177d369ef57231 P 522121848ad548a49cf2f2ee25898ec3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 22:22:24.650710 22135 ts_tablet_manager.cc:1397] T ded13fb73b54433b8d177d369ef57231 P 3e22dfd749ff4f02b4c9ac55e651447e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260808 22:22:24.650861 22144 tablet_replica.cc:439] T ded13fb73b54433b8d177d369ef57231 P 522121848ad548a49cf2f2ee25898ec3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b49d3ab4f0c4429907d17e534a82add" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44137 } } peers { permanent_uuid: "3e22dfd749ff4f02b4c9ac55e651447e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36787 } } peers { permanent_uuid: "522121848ad548a49cf2f2ee25898ec3" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 39019 } } }
I20260808 22:22:24.651005 22144 tablet_replica.cc:442] T ded13fb73b54433b8d177d369ef57231 P 522121848ad548a49cf2f2ee25898ec3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 22:22:24.651060 22135 raft_consensus.cc:348] T ded13fb73b54433b8d177d369ef57231 P 3e22dfd749ff4f02b4c9ac55e651447e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b49d3ab4f0c4429907d17e534a82add" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44137 } } peers { permanent_uuid: "3e22dfd749ff4f02b4c9ac55e651447e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36787 } } peers { permanent_uuid: "522121848ad548a49cf2f2ee25898ec3" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 39019 } }
I20260808 22:22:24.651168 22135 raft_consensus.cc:374] T ded13fb73b54433b8d177d369ef57231 P 3e22dfd749ff4f02b4c9ac55e651447e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:22:24.651208 22135 raft_consensus.cc:729] T ded13fb73b54433b8d177d369ef57231 P 3e22dfd749ff4f02b4c9ac55e651447e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e22dfd749ff4f02b4c9ac55e651447e, State: Initialized, Role: FOLLOWER
I20260808 22:22:24.651348 22135 consensus_queue.cc:260] T ded13fb73b54433b8d177d369ef57231 P 3e22dfd749ff4f02b4c9ac55e651447e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b49d3ab4f0c4429907d17e534a82add" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44137 } } peers { permanent_uuid: "3e22dfd749ff4f02b4c9ac55e651447e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36787 } } peers { permanent_uuid: "522121848ad548a49cf2f2ee25898ec3" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 39019 } }
I20260808 22:22:24.651598 22135 ts_tablet_manager.cc:1428] T ded13fb73b54433b8d177d369ef57231 P 3e22dfd749ff4f02b4c9ac55e651447e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 22:22:24.652236 22143 tablet_replica.cc:439] T ded13fb73b54433b8d177d369ef57231 P 3e22dfd749ff4f02b4c9ac55e651447e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b49d3ab4f0c4429907d17e534a82add" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44137 } } peers { permanent_uuid: "3e22dfd749ff4f02b4c9ac55e651447e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36787 } } peers { permanent_uuid: "522121848ad548a49cf2f2ee25898ec3" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 39019 } } }
I20260808 22:22:24.652346 22143 tablet_replica.cc:442] T ded13fb73b54433b8d177d369ef57231 P 3e22dfd749ff4f02b4c9ac55e651447e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 22:22:24.686573 22143 raft_consensus.cc:482] T ded13fb73b54433b8d177d369ef57231 P 3e22dfd749ff4f02b4c9ac55e651447e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 22:22:24.686683 22143 raft_consensus.cc:504] T ded13fb73b54433b8d177d369ef57231 P 3e22dfd749ff4f02b4c9ac55e651447e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b49d3ab4f0c4429907d17e534a82add" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44137 } } peers { permanent_uuid: "3e22dfd749ff4f02b4c9ac55e651447e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36787 } } peers { permanent_uuid: "522121848ad548a49cf2f2ee25898ec3" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 39019 } }
I20260808 22:22:24.687039 22143 leader_election.cc:288] T ded13fb73b54433b8d177d369ef57231 P 3e22dfd749ff4f02b4c9ac55e651447e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b49d3ab4f0c4429907d17e534a82add (127.9.176.193:44137), 522121848ad548a49cf2f2ee25898ec3 (127.9.176.195:39019)
I20260808 22:22:24.687424 21558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ded13fb73b54433b8d177d369ef57231" candidate_uuid: "3e22dfd749ff4f02b4c9ac55e651447e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b49d3ab4f0c4429907d17e534a82add" is_pre_election: true
I20260808 22:22:24.687491 22000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ded13fb73b54433b8d177d369ef57231" candidate_uuid: "3e22dfd749ff4f02b4c9ac55e651447e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522121848ad548a49cf2f2ee25898ec3" is_pre_election: true
I20260808 22:22:24.687578 21558 raft_consensus.cc:2454] T ded13fb73b54433b8d177d369ef57231 P 7b49d3ab4f0c4429907d17e534a82add [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e22dfd749ff4f02b4c9ac55e651447e in term 0.
I20260808 22:22:24.687597 22000 raft_consensus.cc:2454] T ded13fb73b54433b8d177d369ef57231 P 522121848ad548a49cf2f2ee25898ec3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e22dfd749ff4f02b4c9ac55e651447e in term 0.
I20260808 22:22:24.687917 21660 leader_election.cc:302] T ded13fb73b54433b8d177d369ef57231 P 3e22dfd749ff4f02b4c9ac55e651447e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3e22dfd749ff4f02b4c9ac55e651447e, 522121848ad548a49cf2f2ee25898ec3, 7b49d3ab4f0c4429907d17e534a82add; no voters:
I20260808 22:22:24.688093 22143 raft_consensus.cc:2781] T ded13fb73b54433b8d177d369ef57231 P 3e22dfd749ff4f02b4c9ac55e651447e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 22:22:24.688158 22143 raft_consensus.cc:482] T ded13fb73b54433b8d177d369ef57231 P 3e22dfd749ff4f02b4c9ac55e651447e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 22:22:24.688185 22143 raft_consensus.cc:3037] T ded13fb73b54433b8d177d369ef57231 P 3e22dfd749ff4f02b4c9ac55e651447e [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:22:24.689284 22143 raft_consensus.cc:504] T ded13fb73b54433b8d177d369ef57231 P 3e22dfd749ff4f02b4c9ac55e651447e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b49d3ab4f0c4429907d17e534a82add" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44137 } } peers { permanent_uuid: "3e22dfd749ff4f02b4c9ac55e651447e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36787 } } peers { permanent_uuid: "522121848ad548a49cf2f2ee25898ec3" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 39019 } }
I20260808 22:22:24.689580 22143 leader_election.cc:288] T ded13fb73b54433b8d177d369ef57231 P 3e22dfd749ff4f02b4c9ac55e651447e [CANDIDATE]: Term 1 election: Requested vote from peers 7b49d3ab4f0c4429907d17e534a82add (127.9.176.193:44137), 522121848ad548a49cf2f2ee25898ec3 (127.9.176.195:39019)
I20260808 22:22:24.689908 22000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ded13fb73b54433b8d177d369ef57231" candidate_uuid: "3e22dfd749ff4f02b4c9ac55e651447e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522121848ad548a49cf2f2ee25898ec3"
I20260808 22:22:24.689916 21558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ded13fb73b54433b8d177d369ef57231" candidate_uuid: "3e22dfd749ff4f02b4c9ac55e651447e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b49d3ab4f0c4429907d17e534a82add"
I20260808 22:22:24.690035 22000 raft_consensus.cc:3037] T ded13fb73b54433b8d177d369ef57231 P 522121848ad548a49cf2f2ee25898ec3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:22:24.690032 21558 raft_consensus.cc:3037] T ded13fb73b54433b8d177d369ef57231 P 7b49d3ab4f0c4429907d17e534a82add [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:22:24.691016 21558 raft_consensus.cc:2454] T ded13fb73b54433b8d177d369ef57231 P 7b49d3ab4f0c4429907d17e534a82add [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e22dfd749ff4f02b4c9ac55e651447e in term 1.
I20260808 22:22:24.691046 22000 raft_consensus.cc:2454] T ded13fb73b54433b8d177d369ef57231 P 522121848ad548a49cf2f2ee25898ec3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e22dfd749ff4f02b4c9ac55e651447e in term 1.
I20260808 22:22:24.691282 21663 leader_election.cc:302] T ded13fb73b54433b8d177d369ef57231 P 3e22dfd749ff4f02b4c9ac55e651447e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e22dfd749ff4f02b4c9ac55e651447e, 7b49d3ab4f0c4429907d17e534a82add; no voters:
I20260808 22:22:24.691465 22143 raft_consensus.cc:2781] T ded13fb73b54433b8d177d369ef57231 P 3e22dfd749ff4f02b4c9ac55e651447e [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:22:24.691555 22143 raft_consensus.cc:686] T ded13fb73b54433b8d177d369ef57231 P 3e22dfd749ff4f02b4c9ac55e651447e [term 1 LEADER]: Becoming Leader. State: Replica: 3e22dfd749ff4f02b4c9ac55e651447e, State: Running, Role: LEADER
I20260808 22:22:24.691686 22143 consensus_queue.cc:237] T ded13fb73b54433b8d177d369ef57231 P 3e22dfd749ff4f02b4c9ac55e651447e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b49d3ab4f0c4429907d17e534a82add" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44137 } } peers { permanent_uuid: "3e22dfd749ff4f02b4c9ac55e651447e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36787 } } peers { permanent_uuid: "522121848ad548a49cf2f2ee25898ec3" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 39019 } }
I20260808 22:22:24.692330 22181 tablet_replica.cc:439] T ded13fb73b54433b8d177d369ef57231 P 3e22dfd749ff4f02b4c9ac55e651447e: TxnStatusTablet state changed. Reason: New leader 3e22dfd749ff4f02b4c9ac55e651447e. Latest consensus state: current_term: 1 leader_uuid: "3e22dfd749ff4f02b4c9ac55e651447e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b49d3ab4f0c4429907d17e534a82add" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44137 } } peers { permanent_uuid: "3e22dfd749ff4f02b4c9ac55e651447e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36787 } } peers { permanent_uuid: "522121848ad548a49cf2f2ee25898ec3" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 39019 } } }
I20260808 22:22:24.692446 22181 tablet_replica.cc:442] T ded13fb73b54433b8d177d369ef57231 P 3e22dfd749ff4f02b4c9ac55e651447e: This TxnStatusTablet replica's current role is: LEADER
I20260808 22:22:24.692725 22197 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260808 22:22:24.692915 22197 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260808 22:22:24.693236 21288 catalog_manager.cc:5495] T ded13fb73b54433b8d177d369ef57231 P 3e22dfd749ff4f02b4c9ac55e651447e reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e22dfd749ff4f02b4c9ac55e651447e (127.9.176.194). New cstate: current_term: 1 leader_uuid: "3e22dfd749ff4f02b4c9ac55e651447e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b49d3ab4f0c4429907d17e534a82add" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44137 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e22dfd749ff4f02b4c9ac55e651447e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36787 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "522121848ad548a49cf2f2ee25898ec3" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 39019 } health_report { overall_health: UNKNOWN } } }
I20260808 22:22:24.712272 22190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:24.708375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.194:36787 (local address 127.0.0.1:48156)
0808 22:22:24.708518 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:24.708532 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:24.708624 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:24.708996 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:24.708998 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:24.709005 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 22:22:24.709100 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:24.709105 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:24.710090 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:24.710093 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:24.710701 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:24.710709 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:24.711235 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:24.711237 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:24.711297 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:24.711340 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 22:22:24.712041 (+ 701us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 22:22:24.712049 (+ 8us) client_negotiation.cc:769] Sending connection context
0808 22:22:24.712086 (+ 37us) client_negotiation.cc:240] Negotiation successful
0808 22:22:24.712164 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260808 22:22:24.712294 22131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:24.708466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48156 (local address 127.9.176.194:36787)
0808 22:22:24.708588 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:24.708592 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:24.708608 (+ 16us) server_negotiation.cc:407] Connection header received
0808 22:22:24.708754 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:24.708757 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:24.708820 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:24.708967 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:22:24.709193 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:24.710002 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:24.710793 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:24.711147 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:24.711242 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:24.711459 (+ 217us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:22:24.711955 (+ 496us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:22:24.712073 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:24.712106 (+ 33us) server_negotiation.cc:299] Negotiation successful
0808 22:22:24.712161 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 22:22:24.714975 22000 raft_consensus.cc:1261] T ded13fb73b54433b8d177d369ef57231 P 522121848ad548a49cf2f2ee25898ec3 [term 1 FOLLOWER]: Refusing update from remote peer 3e22dfd749ff4f02b4c9ac55e651447e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:22:24.715193 21558 raft_consensus.cc:1261] T ded13fb73b54433b8d177d369ef57231 P 7b49d3ab4f0c4429907d17e534a82add [term 1 FOLLOWER]: Refusing update from remote peer 3e22dfd749ff4f02b4c9ac55e651447e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:22:24.715456 22181 consensus_queue.cc:1035] T ded13fb73b54433b8d177d369ef57231 P 3e22dfd749ff4f02b4c9ac55e651447e [LEADER]: Connected to new peer: Peer: permanent_uuid: "522121848ad548a49cf2f2ee25898ec3" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 39019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:22:24.715662 22143 consensus_queue.cc:1035] T ded13fb73b54433b8d177d369ef57231 P 3e22dfd749ff4f02b4c9ac55e651447e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b49d3ab4f0c4429907d17e534a82add" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:22:24.717130 22144 tablet_replica.cc:439] T ded13fb73b54433b8d177d369ef57231 P 522121848ad548a49cf2f2ee25898ec3: TxnStatusTablet state changed. Reason: New leader 3e22dfd749ff4f02b4c9ac55e651447e. Latest consensus state: current_term: 1 leader_uuid: "3e22dfd749ff4f02b4c9ac55e651447e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b49d3ab4f0c4429907d17e534a82add" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44137 } } peers { permanent_uuid: "3e22dfd749ff4f02b4c9ac55e651447e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36787 } } peers { permanent_uuid: "522121848ad548a49cf2f2ee25898ec3" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 39019 } } }
I20260808 22:22:24.717248 22144 tablet_replica.cc:442] T ded13fb73b54433b8d177d369ef57231 P 522121848ad548a49cf2f2ee25898ec3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 22:22:24.717301 22145 tablet_replica.cc:439] T ded13fb73b54433b8d177d369ef57231 P 7b49d3ab4f0c4429907d17e534a82add: TxnStatusTablet state changed. Reason: New leader 3e22dfd749ff4f02b4c9ac55e651447e. Latest consensus state: current_term: 1 leader_uuid: "3e22dfd749ff4f02b4c9ac55e651447e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b49d3ab4f0c4429907d17e534a82add" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44137 } } peers { permanent_uuid: "3e22dfd749ff4f02b4c9ac55e651447e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36787 } } peers { permanent_uuid: "522121848ad548a49cf2f2ee25898ec3" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 39019 } } }
I20260808 22:22:24.717417 22145 tablet_replica.cc:442] T ded13fb73b54433b8d177d369ef57231 P 7b49d3ab4f0c4429907d17e534a82add: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 22:22:24.719159 22143 tablet_replica.cc:439] T ded13fb73b54433b8d177d369ef57231 P 3e22dfd749ff4f02b4c9ac55e651447e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3e22dfd749ff4f02b4c9ac55e651447e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b49d3ab4f0c4429907d17e534a82add" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44137 } } peers { permanent_uuid: "3e22dfd749ff4f02b4c9ac55e651447e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36787 } } peers { permanent_uuid: "522121848ad548a49cf2f2ee25898ec3" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 39019 } } }
I20260808 22:22:24.719264 22143 tablet_replica.cc:442] T ded13fb73b54433b8d177d369ef57231 P 3e22dfd749ff4f02b4c9ac55e651447e: This TxnStatusTablet replica's current role is: LEADER
I20260808 22:22:24.719385 22143 tablet_replica.cc:439] T ded13fb73b54433b8d177d369ef57231 P 3e22dfd749ff4f02b4c9ac55e651447e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3e22dfd749ff4f02b4c9ac55e651447e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b49d3ab4f0c4429907d17e534a82add" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44137 } } peers { permanent_uuid: "3e22dfd749ff4f02b4c9ac55e651447e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36787 } } peers { permanent_uuid: "522121848ad548a49cf2f2ee25898ec3" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 39019 } } }
I20260808 22:22:24.719480 22143 tablet_replica.cc:442] T ded13fb73b54433b8d177d369ef57231 P 3e22dfd749ff4f02b4c9ac55e651447e: This TxnStatusTablet replica's current role is: LEADER
I20260808 22:22:24.720577 22144 tablet_replica.cc:439] T ded13fb73b54433b8d177d369ef57231 P 522121848ad548a49cf2f2ee25898ec3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3e22dfd749ff4f02b4c9ac55e651447e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b49d3ab4f0c4429907d17e534a82add" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44137 } } peers { permanent_uuid: "3e22dfd749ff4f02b4c9ac55e651447e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36787 } } peers { permanent_uuid: "522121848ad548a49cf2f2ee25898ec3" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 39019 } } }
I20260808 22:22:24.720685 22144 tablet_replica.cc:442] T ded13fb73b54433b8d177d369ef57231 P 522121848ad548a49cf2f2ee25898ec3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 22:22:24.721530 22145 tablet_replica.cc:439] T ded13fb73b54433b8d177d369ef57231 P 7b49d3ab4f0c4429907d17e534a82add: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3e22dfd749ff4f02b4c9ac55e651447e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b49d3ab4f0c4429907d17e534a82add" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44137 } } peers { permanent_uuid: "3e22dfd749ff4f02b4c9ac55e651447e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36787 } } peers { permanent_uuid: "522121848ad548a49cf2f2ee25898ec3" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 39019 } } }
I20260808 22:22:24.721628 22145 tablet_replica.cc:442] T ded13fb73b54433b8d177d369ef57231 P 7b49d3ab4f0c4429907d17e534a82add: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 22:22:24.726511 22131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:24.723372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48168 (local address 127.9.176.194:36787)
0808 22:22:24.723493 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:24.723496 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:24.723510 (+ 14us) server_negotiation.cc:407] Connection header received
0808 22:22:24.723635 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:24.723638 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:24.723692 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:24.723764 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:22:24.723988 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:24.724468 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:24.725619 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:24.725854 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:24.725924 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:24.726176 (+ 252us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:22:24.726236 (+ 60us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:22:24.726281 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:24.726375 (+ 94us) server_negotiation.cc:299] Negotiation successful
0808 22:22:24.726411 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260808 22:22:24.736126 22131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:24.732841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.194:40915 (local address 127.9.176.194:36787)
0808 22:22:24.732960 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:24.732964 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:24.733208 (+ 244us) server_negotiation.cc:407] Connection header received
0808 22:22:24.733310 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:24.733312 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:24.733348 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:24.733403 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:22:24.733633 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:24.734085 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:24.735251 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:24.735441 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:24.735504 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:24.735778 (+ 274us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:22:24.735849 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:22:24.735897 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:24.735983 (+ 86us) server_negotiation.cc:299] Negotiation successful
0808 22:22:24.736019 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260808 22:22:24.736253 22208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:24.732737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.194:36787 (local address 127.9.176.194:40915)
0808 22:22:24.733128 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:24.733142 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:24.733236 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:24.733427 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:24.733430 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:24.733437 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 22:22:24.733553 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:24.733559 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:24.734171 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:24.734174 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:24.735155 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:24.735165 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:24.735525 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:24.735529 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:24.735622 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:24.735672 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 22:22:24.735913 (+ 241us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 22:22:24.735918 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 22:22:24.735982 (+ 64us) client_negotiation.cc:240] Negotiation successful
0808 22:22:24.736056 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":94,"threads_started":1}
I20260808 22:22:24.742666 22000 raft_consensus.cc:1261] T 2a7e860cdb784ca4bee6e2744670d977 P 522121848ad548a49cf2f2ee25898ec3 [term 1 FOLLOWER]: Refusing update from remote peer 3e22dfd749ff4f02b4c9ac55e651447e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:22:24.742681 21558 raft_consensus.cc:1261] T 2a7e860cdb784ca4bee6e2744670d977 P 7b49d3ab4f0c4429907d17e534a82add [term 1 FOLLOWER]: Refusing update from remote peer 3e22dfd749ff4f02b4c9ac55e651447e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:22:24.743046 22181 consensus_queue.cc:1035] T 2a7e860cdb784ca4bee6e2744670d977 P 3e22dfd749ff4f02b4c9ac55e651447e [LEADER]: Connected to new peer: Peer: permanent_uuid: "522121848ad548a49cf2f2ee25898ec3" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 39019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:22:24.743213 22204 consensus_queue.cc:1035] T 2a7e860cdb784ca4bee6e2744670d977 P 3e22dfd749ff4f02b4c9ac55e651447e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b49d3ab4f0c4429907d17e534a82add" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:22:24.787390 21287 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58514:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:22:24.787544 21287 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58514:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:22:24.789469 21287 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 824baa547e754adfbe38c0b466a74c12: Sending DeleteTablet for 3 replicas of tablet 2a7e860cdb784ca4bee6e2744670d977
I20260808 22:22:24.789944 21528 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a7e860cdb784ca4bee6e2744670d977 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:22:24 UTC) from {username='oryx', principal='oryx/127.9.176.254@KRBTEST.COM'} at 127.0.0.1:45358
I20260808 22:22:24.790031 21743 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a7e860cdb784ca4bee6e2744670d977 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:22:24 UTC) from {username='oryx', principal='oryx/127.9.176.254@KRBTEST.COM'} at 127.0.0.1:48144
I20260808 22:22:24.790031 21968 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a7e860cdb784ca4bee6e2744670d977 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:22:24 UTC) from {username='oryx', principal='oryx/127.9.176.254@KRBTEST.COM'} at 127.0.0.1:41794
I20260808 22:22:24.790135 22172 tablet_replica.cc:330] T 2a7e860cdb784ca4bee6e2744670d977 P 7b49d3ab4f0c4429907d17e534a82add: stopping tablet replica
I20260808 22:22:24.790198 22175 tablet_replica.cc:330] T 2a7e860cdb784ca4bee6e2744670d977 P 522121848ad548a49cf2f2ee25898ec3: stopping tablet replica
I20260808 22:22:24.790246 22172 raft_consensus.cc:2229] T 2a7e860cdb784ca4bee6e2744670d977 P 7b49d3ab4f0c4429907d17e534a82add [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:22:24.790300 22175 raft_consensus.cc:2229] T 2a7e860cdb784ca4bee6e2744670d977 P 522121848ad548a49cf2f2ee25898ec3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:22:24.790349 22172 raft_consensus.cc:2258] T 2a7e860cdb784ca4bee6e2744670d977 P 7b49d3ab4f0c4429907d17e534a82add [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:22:24.790410 22175 raft_consensus.cc:2258] T 2a7e860cdb784ca4bee6e2744670d977 P 522121848ad548a49cf2f2ee25898ec3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:22:24.790508 22170 tablet_replica.cc:330] T 2a7e860cdb784ca4bee6e2744670d977 P 3e22dfd749ff4f02b4c9ac55e651447e: stopping tablet replica
I20260808 22:22:24.790611 22170 raft_consensus.cc:2229] T 2a7e860cdb784ca4bee6e2744670d977 P 3e22dfd749ff4f02b4c9ac55e651447e [term 1 LEADER]: Raft consensus shutting down.
I20260808 22:22:24.790866 22170 raft_consensus.cc:2258] T 2a7e860cdb784ca4bee6e2744670d977 P 3e22dfd749ff4f02b4c9ac55e651447e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:22:24.792060 22172 ts_tablet_manager.cc:1905] T 2a7e860cdb784ca4bee6e2744670d977 P 7b49d3ab4f0c4429907d17e534a82add: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:22:24.794924 22172 ts_tablet_manager.cc:1918] T 2a7e860cdb784ca4bee6e2744670d977 P 7b49d3ab4f0c4429907d17e534a82add: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 22:22:24.794994 22172 log.cc:1196] T 2a7e860cdb784ca4bee6e2744670d977 P 7b49d3ab4f0c4429907d17e534a82add: Deleting WAL directory at /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/minicluster-data/ts-0/wal/wals/2a7e860cdb784ca4bee6e2744670d977
I20260808 22:22:24.795254 22172 ts_tablet_manager.cc:1939] T 2a7e860cdb784ca4bee6e2744670d977 P 7b49d3ab4f0c4429907d17e534a82add: Deleting consensus metadata
I20260808 22:22:24.795938 21258 catalog_manager.cc:4841] TS 7b49d3ab4f0c4429907d17e534a82add (127.9.176.193:44137): tablet 2a7e860cdb784ca4bee6e2744670d977 (table test-table [id=459b6357f5654d77aea834d11e1cd95f]) successfully deleted
I20260808 22:22:24.796509 22170 ts_tablet_manager.cc:1905] T 2a7e860cdb784ca4bee6e2744670d977 P 3e22dfd749ff4f02b4c9ac55e651447e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:22:24.799154 22170 ts_tablet_manager.cc:1918] T 2a7e860cdb784ca4bee6e2744670d977 P 3e22dfd749ff4f02b4c9ac55e651447e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 22:22:24.799232 22170 log.cc:1196] T 2a7e860cdb784ca4bee6e2744670d977 P 3e22dfd749ff4f02b4c9ac55e651447e: Deleting WAL directory at /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/minicluster-data/ts-1/wal/wals/2a7e860cdb784ca4bee6e2744670d977
I20260808 22:22:24.799481 22170 ts_tablet_manager.cc:1939] T 2a7e860cdb784ca4bee6e2744670d977 P 3e22dfd749ff4f02b4c9ac55e651447e: Deleting consensus metadata
I20260808 22:22:24.800092 21256 catalog_manager.cc:4841] TS 3e22dfd749ff4f02b4c9ac55e651447e (127.9.176.194:36787): tablet 2a7e860cdb784ca4bee6e2744670d977 (table test-table [id=459b6357f5654d77aea834d11e1cd95f]) successfully deleted
I20260808 22:22:24.801075 22175 ts_tablet_manager.cc:1905] T 2a7e860cdb784ca4bee6e2744670d977 P 522121848ad548a49cf2f2ee25898ec3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:22:24.803614 22175 ts_tablet_manager.cc:1918] T 2a7e860cdb784ca4bee6e2744670d977 P 522121848ad548a49cf2f2ee25898ec3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 22:22:24.803709 22175 log.cc:1196] T 2a7e860cdb784ca4bee6e2744670d977 P 522121848ad548a49cf2f2ee25898ec3: Deleting WAL directory at /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/minicluster-data/ts-2/wal/wals/2a7e860cdb784ca4bee6e2744670d977
I20260808 22:22:24.803948 22175 ts_tablet_manager.cc:1939] T 2a7e860cdb784ca4bee6e2744670d977 P 522121848ad548a49cf2f2ee25898ec3: Deleting consensus metadata
I20260808 22:22:24.804603 21257 catalog_manager.cc:4841] TS 522121848ad548a49cf2f2ee25898ec3 (127.9.176.195:39019): tablet 2a7e860cdb784ca4bee6e2744670d977 (table test-table [id=459b6357f5654d77aea834d11e1cd95f]) successfully deleted
Aug 08 22:22:24 dist-test-slave-ncp9 krb5kdc[21216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 22:22:24 dist-test-slave-ncp9 krb5kdc[21216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:22:24 dist-test-slave-ncp9 krb5kdc[21216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
W20260808 22:22:24.822604 21373 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:22:24.816186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58540 (local address 127.9.176.254:46215)
0808 22:22:24.816304 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:24.816308 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:24.816504 (+ 196us) server_negotiation.cc:407] Connection header received
0808 22:22:24.816634 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:24.816637 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:24.816693 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:24.816759 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:24.817528 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:24.818103 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:24.819020 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:24.819372 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:24.819479 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:24.822480 (+ 3001us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58540: BlockingRecv error: recv got EOF from 127.0.0.1:58540 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Aug 08 22:22:24 dist-test-slave-ncp9 krb5kdc[21216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM
I20260808 22:22:24.832186 21373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:24.824102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58556 (local address 127.9.176.254:46215)
0808 22:22:24.824209 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:24.824213 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:24.824223 (+ 10us) server_negotiation.cc:407] Connection header received
0808 22:22:24.824253 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:24.824255 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:24.824289 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:24.824350 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:24.825061 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:24.825573 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:24.826632 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:24.826853 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:24.826923 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:24.829474 (+ 2551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:24.829477 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:24.829479 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:22:24.829503 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:24.830748 (+ 1245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:24.831191 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:24.831193 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:24.831195 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:24.831259 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:24.831491 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:24.831493 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:24.831494 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:24.831650 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:24.831744 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:24.831967 (+ 223us) server_negotiation.cc:299] Negotiation successful
0808 22:22:24.832052 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260808 22:22:24.834635 21287 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58556:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 22:22:24.835122 21287 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 22:22:24.840070 21528 tablet_service.cc:1467] Processing CreateTablet for tablet 8884ff46281b44d2b8c20530d7789bdb (DEFAULT_TABLE table=test-table [id=ca51a016508d42a89e2b02acb6effe2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:22:24.840310 21968 tablet_service.cc:1467] Processing CreateTablet for tablet 8884ff46281b44d2b8c20530d7789bdb (DEFAULT_TABLE table=test-table [id=ca51a016508d42a89e2b02acb6effe2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:22:24.840405 21528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8884ff46281b44d2b8c20530d7789bdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:22:24.840603 21968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8884ff46281b44d2b8c20530d7789bdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:22:24.842777 21743 tablet_service.cc:1467] Processing CreateTablet for tablet 8884ff46281b44d2b8c20530d7789bdb (DEFAULT_TABLE table=test-table [id=ca51a016508d42a89e2b02acb6effe2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:22:24.843015 22140 tablet_bootstrap.cc:492] T 8884ff46281b44d2b8c20530d7789bdb P 7b49d3ab4f0c4429907d17e534a82add: Bootstrap starting.
I20260808 22:22:24.843060 21743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8884ff46281b44d2b8c20530d7789bdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:22:24.843914 22140 tablet_bootstrap.cc:654] T 8884ff46281b44d2b8c20530d7789bdb P 7b49d3ab4f0c4429907d17e534a82add: Neither blocks nor log segments found. Creating new log.
I20260808 22:22:24.844877 22135 tablet_bootstrap.cc:492] T 8884ff46281b44d2b8c20530d7789bdb P 3e22dfd749ff4f02b4c9ac55e651447e: Bootstrap starting.
I20260808 22:22:24.845036 22136 tablet_bootstrap.cc:492] T 8884ff46281b44d2b8c20530d7789bdb P 522121848ad548a49cf2f2ee25898ec3: Bootstrap starting.
I20260808 22:22:24.845254 22140 tablet_bootstrap.cc:492] T 8884ff46281b44d2b8c20530d7789bdb P 7b49d3ab4f0c4429907d17e534a82add: No bootstrap required, opened a new log
I20260808 22:22:24.845333 22140 ts_tablet_manager.cc:1397] T 8884ff46281b44d2b8c20530d7789bdb P 7b49d3ab4f0c4429907d17e534a82add: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 22:22:24.845829 22135 tablet_bootstrap.cc:654] T 8884ff46281b44d2b8c20530d7789bdb P 3e22dfd749ff4f02b4c9ac55e651447e: Neither blocks nor log segments found. Creating new log.
I20260808 22:22:24.845829 22136 tablet_bootstrap.cc:654] T 8884ff46281b44d2b8c20530d7789bdb P 522121848ad548a49cf2f2ee25898ec3: Neither blocks nor log segments found. Creating new log.
I20260808 22:22:24.845736 22140 raft_consensus.cc:348] T 8884ff46281b44d2b8c20530d7789bdb P 7b49d3ab4f0c4429907d17e534a82add [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b49d3ab4f0c4429907d17e534a82add" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44137 } } peers { permanent_uuid: "3e22dfd749ff4f02b4c9ac55e651447e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36787 } } peers { permanent_uuid: "522121848ad548a49cf2f2ee25898ec3" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 39019 } }
I20260808 22:22:24.845875 22140 raft_consensus.cc:374] T 8884ff46281b44d2b8c20530d7789bdb P 7b49d3ab4f0c4429907d17e534a82add [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:22:24.845913 22140 raft_consensus.cc:729] T 8884ff46281b44d2b8c20530d7789bdb P 7b49d3ab4f0c4429907d17e534a82add [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b49d3ab4f0c4429907d17e534a82add, State: Initialized, Role: FOLLOWER
I20260808 22:22:24.846048 22140 consensus_queue.cc:260] T 8884ff46281b44d2b8c20530d7789bdb P 7b49d3ab4f0c4429907d17e534a82add [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b49d3ab4f0c4429907d17e534a82add" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44137 } } peers { permanent_uuid: "3e22dfd749ff4f02b4c9ac55e651447e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36787 } } peers { permanent_uuid: "522121848ad548a49cf2f2ee25898ec3" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 39019 } }
I20260808 22:22:24.846318 22140 ts_tablet_manager.cc:1428] T 8884ff46281b44d2b8c20530d7789bdb P 7b49d3ab4f0c4429907d17e534a82add: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 22:22:24.848013 22136 tablet_bootstrap.cc:492] T 8884ff46281b44d2b8c20530d7789bdb P 522121848ad548a49cf2f2ee25898ec3: No bootstrap required, opened a new log
I20260808 22:22:24.848100 22136 ts_tablet_manager.cc:1397] T 8884ff46281b44d2b8c20530d7789bdb P 522121848ad548a49cf2f2ee25898ec3: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260808 22:22:24.848472 22136 raft_consensus.cc:348] T 8884ff46281b44d2b8c20530d7789bdb P 522121848ad548a49cf2f2ee25898ec3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b49d3ab4f0c4429907d17e534a82add" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44137 } } peers { permanent_uuid: "3e22dfd749ff4f02b4c9ac55e651447e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36787 } } peers { permanent_uuid: "522121848ad548a49cf2f2ee25898ec3" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 39019 } }
I20260808 22:22:24.848568 22136 raft_consensus.cc:374] T 8884ff46281b44d2b8c20530d7789bdb P 522121848ad548a49cf2f2ee25898ec3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:22:24.848598 22136 raft_consensus.cc:729] T 8884ff46281b44d2b8c20530d7789bdb P 522121848ad548a49cf2f2ee25898ec3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522121848ad548a49cf2f2ee25898ec3, State: Initialized, Role: FOLLOWER
I20260808 22:22:24.848714 22136 consensus_queue.cc:260] T 8884ff46281b44d2b8c20530d7789bdb P 522121848ad548a49cf2f2ee25898ec3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b49d3ab4f0c4429907d17e534a82add" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44137 } } peers { permanent_uuid: "3e22dfd749ff4f02b4c9ac55e651447e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36787 } } peers { permanent_uuid: "522121848ad548a49cf2f2ee25898ec3" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 39019 } }
I20260808 22:22:24.849538 22136 ts_tablet_manager.cc:1428] T 8884ff46281b44d2b8c20530d7789bdb P 522121848ad548a49cf2f2ee25898ec3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 22:22:24.850430 22135 tablet_bootstrap.cc:492] T 8884ff46281b44d2b8c20530d7789bdb P 3e22dfd749ff4f02b4c9ac55e651447e: No bootstrap required, opened a new log
I20260808 22:22:24.850533 22135 ts_tablet_manager.cc:1397] T 8884ff46281b44d2b8c20530d7789bdb P 3e22dfd749ff4f02b4c9ac55e651447e: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20260808 22:22:24.850939 22135 raft_consensus.cc:348] T 8884ff46281b44d2b8c20530d7789bdb P 3e22dfd749ff4f02b4c9ac55e651447e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b49d3ab4f0c4429907d17e534a82add" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44137 } } peers { permanent_uuid: "3e22dfd749ff4f02b4c9ac55e651447e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36787 } } peers { permanent_uuid: "522121848ad548a49cf2f2ee25898ec3" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 39019 } }
I20260808 22:22:24.851059 22135 raft_consensus.cc:374] T 8884ff46281b44d2b8c20530d7789bdb P 3e22dfd749ff4f02b4c9ac55e651447e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:22:24.851089 22135 raft_consensus.cc:729] T 8884ff46281b44d2b8c20530d7789bdb P 3e22dfd749ff4f02b4c9ac55e651447e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e22dfd749ff4f02b4c9ac55e651447e, State: Initialized, Role: FOLLOWER
I20260808 22:22:24.851207 22135 consensus_queue.cc:260] T 8884ff46281b44d2b8c20530d7789bdb P 3e22dfd749ff4f02b4c9ac55e651447e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b49d3ab4f0c4429907d17e534a82add" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44137 } } peers { permanent_uuid: "3e22dfd749ff4f02b4c9ac55e651447e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36787 } } peers { permanent_uuid: "522121848ad548a49cf2f2ee25898ec3" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 39019 } }
I20260808 22:22:24.851420 22135 ts_tablet_manager.cc:1428] T 8884ff46281b44d2b8c20530d7789bdb P 3e22dfd749ff4f02b4c9ac55e651447e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260808 22:22:24.886314 22204 raft_consensus.cc:482] T 8884ff46281b44d2b8c20530d7789bdb P 3e22dfd749ff4f02b4c9ac55e651447e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 22:22:24.886440 22204 raft_consensus.cc:504] T 8884ff46281b44d2b8c20530d7789bdb P 3e22dfd749ff4f02b4c9ac55e651447e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b49d3ab4f0c4429907d17e534a82add" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44137 } } peers { permanent_uuid: "3e22dfd749ff4f02b4c9ac55e651447e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36787 } } peers { permanent_uuid: "522121848ad548a49cf2f2ee25898ec3" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 39019 } }
I20260808 22:22:24.886832 22204 leader_election.cc:288] T 8884ff46281b44d2b8c20530d7789bdb P 3e22dfd749ff4f02b4c9ac55e651447e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b49d3ab4f0c4429907d17e534a82add (127.9.176.193:44137), 522121848ad548a49cf2f2ee25898ec3 (127.9.176.195:39019)
I20260808 22:22:24.887233 21556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8884ff46281b44d2b8c20530d7789bdb" candidate_uuid: "3e22dfd749ff4f02b4c9ac55e651447e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b49d3ab4f0c4429907d17e534a82add" is_pre_election: true
I20260808 22:22:24.887223 22000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8884ff46281b44d2b8c20530d7789bdb" candidate_uuid: "3e22dfd749ff4f02b4c9ac55e651447e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522121848ad548a49cf2f2ee25898ec3" is_pre_election: true
I20260808 22:22:24.887370 22000 raft_consensus.cc:2454] T 8884ff46281b44d2b8c20530d7789bdb P 522121848ad548a49cf2f2ee25898ec3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e22dfd749ff4f02b4c9ac55e651447e in term 0.
I20260808 22:22:24.887374 21556 raft_consensus.cc:2454] T 8884ff46281b44d2b8c20530d7789bdb P 7b49d3ab4f0c4429907d17e534a82add [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e22dfd749ff4f02b4c9ac55e651447e in term 0.
I20260808 22:22:24.887640 21660 leader_election.cc:302] T 8884ff46281b44d2b8c20530d7789bdb P 3e22dfd749ff4f02b4c9ac55e651447e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e22dfd749ff4f02b4c9ac55e651447e, 522121848ad548a49cf2f2ee25898ec3; no voters:
I20260808 22:22:24.887813 22204 raft_consensus.cc:2781] T 8884ff46281b44d2b8c20530d7789bdb P 3e22dfd749ff4f02b4c9ac55e651447e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 22:22:24.887882 22204 raft_consensus.cc:482] T 8884ff46281b44d2b8c20530d7789bdb P 3e22dfd749ff4f02b4c9ac55e651447e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 22:22:24.887923 22204 raft_consensus.cc:3037] T 8884ff46281b44d2b8c20530d7789bdb P 3e22dfd749ff4f02b4c9ac55e651447e [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:22:24.889173 22204 raft_consensus.cc:504] T 8884ff46281b44d2b8c20530d7789bdb P 3e22dfd749ff4f02b4c9ac55e651447e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b49d3ab4f0c4429907d17e534a82add" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44137 } } peers { permanent_uuid: "3e22dfd749ff4f02b4c9ac55e651447e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36787 } } peers { permanent_uuid: "522121848ad548a49cf2f2ee25898ec3" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 39019 } }
I20260808 22:22:24.889505 22204 leader_election.cc:288] T 8884ff46281b44d2b8c20530d7789bdb P 3e22dfd749ff4f02b4c9ac55e651447e [CANDIDATE]: Term 1 election: Requested vote from peers 7b49d3ab4f0c4429907d17e534a82add (127.9.176.193:44137), 522121848ad548a49cf2f2ee25898ec3 (127.9.176.195:39019)
I20260808 22:22:24.889801 21556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8884ff46281b44d2b8c20530d7789bdb" candidate_uuid: "3e22dfd749ff4f02b4c9ac55e651447e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b49d3ab4f0c4429907d17e534a82add"
I20260808 22:22:24.889832 22000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8884ff46281b44d2b8c20530d7789bdb" candidate_uuid: "3e22dfd749ff4f02b4c9ac55e651447e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522121848ad548a49cf2f2ee25898ec3"
I20260808 22:22:24.889918 21556 raft_consensus.cc:3037] T 8884ff46281b44d2b8c20530d7789bdb P 7b49d3ab4f0c4429907d17e534a82add [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:22:24.889931 22000 raft_consensus.cc:3037] T 8884ff46281b44d2b8c20530d7789bdb P 522121848ad548a49cf2f2ee25898ec3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:22:24.890831 21556 raft_consensus.cc:2454] T 8884ff46281b44d2b8c20530d7789bdb P 7b49d3ab4f0c4429907d17e534a82add [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e22dfd749ff4f02b4c9ac55e651447e in term 1.
I20260808 22:22:24.890981 22000 raft_consensus.cc:2454] T 8884ff46281b44d2b8c20530d7789bdb P 522121848ad548a49cf2f2ee25898ec3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e22dfd749ff4f02b4c9ac55e651447e in term 1.
I20260808 22:22:24.891109 21663 leader_election.cc:302] T 8884ff46281b44d2b8c20530d7789bdb P 3e22dfd749ff4f02b4c9ac55e651447e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e22dfd749ff4f02b4c9ac55e651447e, 7b49d3ab4f0c4429907d17e534a82add; no voters:
I20260808 22:22:24.891398 22204 raft_consensus.cc:2781] T 8884ff46281b44d2b8c20530d7789bdb P 3e22dfd749ff4f02b4c9ac55e651447e [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:22:24.891492 22204 raft_consensus.cc:686] T 8884ff46281b44d2b8c20530d7789bdb P 3e22dfd749ff4f02b4c9ac55e651447e [term 1 LEADER]: Becoming Leader. State: Replica: 3e22dfd749ff4f02b4c9ac55e651447e, State: Running, Role: LEADER
I20260808 22:22:24.891623 22204 consensus_queue.cc:237] T 8884ff46281b44d2b8c20530d7789bdb P 3e22dfd749ff4f02b4c9ac55e651447e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b49d3ab4f0c4429907d17e534a82add" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44137 } } peers { permanent_uuid: "3e22dfd749ff4f02b4c9ac55e651447e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36787 } } peers { permanent_uuid: "522121848ad548a49cf2f2ee25898ec3" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 39019 } }
I20260808 22:22:24.893260 21286 catalog_manager.cc:5495] T 8884ff46281b44d2b8c20530d7789bdb P 3e22dfd749ff4f02b4c9ac55e651447e reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e22dfd749ff4f02b4c9ac55e651447e (127.9.176.194). New cstate: current_term: 1 leader_uuid: "3e22dfd749ff4f02b4c9ac55e651447e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b49d3ab4f0c4429907d17e534a82add" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44137 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e22dfd749ff4f02b4c9ac55e651447e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36787 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "522121848ad548a49cf2f2ee25898ec3" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 39019 } health_report { overall_health: UNKNOWN } } }
I20260808 22:22:24.910434 22131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:24.907306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48182 (local address 127.9.176.194:36787)
0808 22:22:24.907407 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:24.907410 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:24.907532 (+ 122us) server_negotiation.cc:407] Connection header received
0808 22:22:24.907683 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:24.907687 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:24.907732 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:24.907792 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:22:24.908037 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:24.908576 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:24.909474 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:24.909699 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:24.909785 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:24.910034 (+ 249us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:22:24.910104 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:22:24.910160 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:24.910258 (+ 98us) server_negotiation.cc:299] Negotiation successful
0808 22:22:24.910306 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260808 22:22:24.912076 21556 raft_consensus.cc:1261] T 8884ff46281b44d2b8c20530d7789bdb P 7b49d3ab4f0c4429907d17e534a82add [term 1 FOLLOWER]: Refusing update from remote peer 3e22dfd749ff4f02b4c9ac55e651447e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:22:24.912096 22000 raft_consensus.cc:1261] T 8884ff46281b44d2b8c20530d7789bdb P 522121848ad548a49cf2f2ee25898ec3 [term 1 FOLLOWER]: Refusing update from remote peer 3e22dfd749ff4f02b4c9ac55e651447e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:22:24.912487 22143 consensus_queue.cc:1035] T 8884ff46281b44d2b8c20530d7789bdb P 3e22dfd749ff4f02b4c9ac55e651447e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b49d3ab4f0c4429907d17e534a82add" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:22:24.912629 22181 consensus_queue.cc:1035] T 8884ff46281b44d2b8c20530d7789bdb P 3e22dfd749ff4f02b4c9ac55e651447e [LEADER]: Connected to new peer: Peer: permanent_uuid: "522121848ad548a49cf2f2ee25898ec3" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 39019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:22:24.920987 21287 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58556:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:22:24.921095 21287 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58556:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:22:24.922919 21287 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 824baa547e754adfbe38c0b466a74c12: Sending DeleteTablet for 3 replicas of tablet 8884ff46281b44d2b8c20530d7789bdb
I20260808 22:22:24.923420 21528 tablet_service.cc:1514] Processing DeleteTablet for tablet 8884ff46281b44d2b8c20530d7789bdb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:22:24 UTC) from {username='oryx', principal='oryx/127.9.176.254@KRBTEST.COM'} at 127.0.0.1:45358
I20260808 22:22:24.923445 21743 tablet_service.cc:1514] Processing DeleteTablet for tablet 8884ff46281b44d2b8c20530d7789bdb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:22:24 UTC) from {username='oryx', principal='oryx/127.9.176.254@KRBTEST.COM'} at 127.0.0.1:48144
I20260808 22:22:24.923614 22172 tablet_replica.cc:330] T 8884ff46281b44d2b8c20530d7789bdb P 7b49d3ab4f0c4429907d17e534a82add: stopping tablet replica
I20260808 22:22:24.923727 22172 raft_consensus.cc:2229] T 8884ff46281b44d2b8c20530d7789bdb P 7b49d3ab4f0c4429907d17e534a82add [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:22:24.923832 22172 raft_consensus.cc:2258] T 8884ff46281b44d2b8c20530d7789bdb P 7b49d3ab4f0c4429907d17e534a82add [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:22:24.923978 22170 tablet_replica.cc:330] T 8884ff46281b44d2b8c20530d7789bdb P 3e22dfd749ff4f02b4c9ac55e651447e: stopping tablet replica
I20260808 22:22:24.923976 21968 tablet_service.cc:1514] Processing DeleteTablet for tablet 8884ff46281b44d2b8c20530d7789bdb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:22:24 UTC) from {username='oryx', principal='oryx/127.9.176.254@KRBTEST.COM'} at 127.0.0.1:41794
I20260808 22:22:24.924072 22170 raft_consensus.cc:2229] T 8884ff46281b44d2b8c20530d7789bdb P 3e22dfd749ff4f02b4c9ac55e651447e [term 1 LEADER]: Raft consensus shutting down.
I20260808 22:22:24.924137 22175 tablet_replica.cc:330] T 8884ff46281b44d2b8c20530d7789bdb P 522121848ad548a49cf2f2ee25898ec3: stopping tablet replica
I20260808 22:22:24.924212 22170 raft_consensus.cc:2258] T 8884ff46281b44d2b8c20530d7789bdb P 3e22dfd749ff4f02b4c9ac55e651447e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:22:24.924247 22175 raft_consensus.cc:2229] T 8884ff46281b44d2b8c20530d7789bdb P 522121848ad548a49cf2f2ee25898ec3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:22:24.924387 22175 raft_consensus.cc:2258] T 8884ff46281b44d2b8c20530d7789bdb P 522121848ad548a49cf2f2ee25898ec3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:22:24.925153 22170 ts_tablet_manager.cc:1905] T 8884ff46281b44d2b8c20530d7789bdb P 3e22dfd749ff4f02b4c9ac55e651447e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:22:24.925227 22175 ts_tablet_manager.cc:1905] T 8884ff46281b44d2b8c20530d7789bdb P 522121848ad548a49cf2f2ee25898ec3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:22:24.926950 22175 ts_tablet_manager.cc:1918] T 8884ff46281b44d2b8c20530d7789bdb P 522121848ad548a49cf2f2ee25898ec3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 22:22:24.927001 22175 log.cc:1196] T 8884ff46281b44d2b8c20530d7789bdb P 522121848ad548a49cf2f2ee25898ec3: Deleting WAL directory at /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/minicluster-data/ts-2/wal/wals/8884ff46281b44d2b8c20530d7789bdb
I20260808 22:22:24.927210 22175 ts_tablet_manager.cc:1939] T 8884ff46281b44d2b8c20530d7789bdb P 522121848ad548a49cf2f2ee25898ec3: Deleting consensus metadata
I20260808 22:22:24.927476 22170 ts_tablet_manager.cc:1918] T 8884ff46281b44d2b8c20530d7789bdb P 3e22dfd749ff4f02b4c9ac55e651447e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 22:22:24.927541 22170 log.cc:1196] T 8884ff46281b44d2b8c20530d7789bdb P 3e22dfd749ff4f02b4c9ac55e651447e: Deleting WAL directory at /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/minicluster-data/ts-1/wal/wals/8884ff46281b44d2b8c20530d7789bdb
I20260808 22:22:24.927670 22172 ts_tablet_manager.cc:1905] T 8884ff46281b44d2b8c20530d7789bdb P 7b49d3ab4f0c4429907d17e534a82add: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:22:24.927773 22170 ts_tablet_manager.cc:1939] T 8884ff46281b44d2b8c20530d7789bdb P 3e22dfd749ff4f02b4c9ac55e651447e: Deleting consensus metadata
I20260808 22:22:24.928352 21256 catalog_manager.cc:4841] TS 3e22dfd749ff4f02b4c9ac55e651447e (127.9.176.194:36787): tablet 8884ff46281b44d2b8c20530d7789bdb (table test-table [id=ca51a016508d42a89e2b02acb6effe2d]) successfully deleted
I20260808 22:22:24.929395 21257 catalog_manager.cc:4841] TS 522121848ad548a49cf2f2ee25898ec3 (127.9.176.195:39019): tablet 8884ff46281b44d2b8c20530d7789bdb (table test-table [id=ca51a016508d42a89e2b02acb6effe2d]) successfully deleted
I20260808 22:22:24.929389 21287 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58556:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260808 22:22:24.929802 22172 ts_tablet_manager.cc:1918] T 8884ff46281b44d2b8c20530d7789bdb P 7b49d3ab4f0c4429907d17e534a82add: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260808 22:22:24.929848 21287 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 22:22:24.929869 22172 log.cc:1196] T 8884ff46281b44d2b8c20530d7789bdb P 7b49d3ab4f0c4429907d17e534a82add: Deleting WAL directory at /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/minicluster-data/ts-0/wal/wals/8884ff46281b44d2b8c20530d7789bdb
I20260808 22:22:24.930083 22172 ts_tablet_manager.cc:1939] T 8884ff46281b44d2b8c20530d7789bdb P 7b49d3ab4f0c4429907d17e534a82add: Deleting consensus metadata
I20260808 22:22:24.930624 21258 catalog_manager.cc:4841] TS 7b49d3ab4f0c4429907d17e534a82add (127.9.176.193:44137): tablet 8884ff46281b44d2b8c20530d7789bdb (table test-table [id=ca51a016508d42a89e2b02acb6effe2d]) successfully deleted
I20260808 22:22:24.934199 21968 tablet_service.cc:1467] Processing CreateTablet for tablet 02ea94ba0e3d4970a9c6c9729b15b785 (DEFAULT_TABLE table=test-table [id=24e3bddce6994c9cbe7bf49b64f6cb67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:22:24.934218 21743 tablet_service.cc:1467] Processing CreateTablet for tablet 02ea94ba0e3d4970a9c6c9729b15b785 (DEFAULT_TABLE table=test-table [id=24e3bddce6994c9cbe7bf49b64f6cb67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:22:24.934461 21743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02ea94ba0e3d4970a9c6c9729b15b785. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:22:24.934502 21968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02ea94ba0e3d4970a9c6c9729b15b785. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:22:24.936160 22136 tablet_bootstrap.cc:492] T 02ea94ba0e3d4970a9c6c9729b15b785 P 522121848ad548a49cf2f2ee25898ec3: Bootstrap starting.
I20260808 22:22:24.936537 21528 tablet_service.cc:1467] Processing CreateTablet for tablet 02ea94ba0e3d4970a9c6c9729b15b785 (DEFAULT_TABLE table=test-table [id=24e3bddce6994c9cbe7bf49b64f6cb67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:22:24.936769 21528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02ea94ba0e3d4970a9c6c9729b15b785. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:22:24.936786 22135 tablet_bootstrap.cc:492] T 02ea94ba0e3d4970a9c6c9729b15b785 P 3e22dfd749ff4f02b4c9ac55e651447e: Bootstrap starting.
I20260808 22:22:24.937232 22136 tablet_bootstrap.cc:654] T 02ea94ba0e3d4970a9c6c9729b15b785 P 522121848ad548a49cf2f2ee25898ec3: Neither blocks nor log segments found. Creating new log.
I20260808 22:22:24.937565 22135 tablet_bootstrap.cc:654] T 02ea94ba0e3d4970a9c6c9729b15b785 P 3e22dfd749ff4f02b4c9ac55e651447e: Neither blocks nor log segments found. Creating new log.
I20260808 22:22:24.938915 22135 tablet_bootstrap.cc:492] T 02ea94ba0e3d4970a9c6c9729b15b785 P 3e22dfd749ff4f02b4c9ac55e651447e: No bootstrap required, opened a new log
I20260808 22:22:24.939015 22135 ts_tablet_manager.cc:1397] T 02ea94ba0e3d4970a9c6c9729b15b785 P 3e22dfd749ff4f02b4c9ac55e651447e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 22:22:24.939330 22136 tablet_bootstrap.cc:492] T 02ea94ba0e3d4970a9c6c9729b15b785 P 522121848ad548a49cf2f2ee25898ec3: No bootstrap required, opened a new log
I20260808 22:22:24.939409 22136 ts_tablet_manager.cc:1397] T 02ea94ba0e3d4970a9c6c9729b15b785 P 522121848ad548a49cf2f2ee25898ec3: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260808 22:22:24.939407 22135 raft_consensus.cc:348] T 02ea94ba0e3d4970a9c6c9729b15b785 P 3e22dfd749ff4f02b4c9ac55e651447e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b49d3ab4f0c4429907d17e534a82add" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44137 } } peers { permanent_uuid: "3e22dfd749ff4f02b4c9ac55e651447e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36787 } } peers { permanent_uuid: "522121848ad548a49cf2f2ee25898ec3" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 39019 } }
I20260808 22:22:24.939519 22135 raft_consensus.cc:374] T 02ea94ba0e3d4970a9c6c9729b15b785 P 3e22dfd749ff4f02b4c9ac55e651447e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:22:24.939556 22135 raft_consensus.cc:729] T 02ea94ba0e3d4970a9c6c9729b15b785 P 3e22dfd749ff4f02b4c9ac55e651447e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e22dfd749ff4f02b4c9ac55e651447e, State: Initialized, Role: FOLLOWER
I20260808 22:22:24.939672 22135 consensus_queue.cc:260] T 02ea94ba0e3d4970a9c6c9729b15b785 P 3e22dfd749ff4f02b4c9ac55e651447e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b49d3ab4f0c4429907d17e534a82add" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44137 } } peers { permanent_uuid: "3e22dfd749ff4f02b4c9ac55e651447e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36787 } } peers { permanent_uuid: "522121848ad548a49cf2f2ee25898ec3" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 39019 } }
I20260808 22:22:24.939833 22136 raft_consensus.cc:348] T 02ea94ba0e3d4970a9c6c9729b15b785 P 522121848ad548a49cf2f2ee25898ec3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b49d3ab4f0c4429907d17e534a82add" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44137 } } peers { permanent_uuid: "3e22dfd749ff4f02b4c9ac55e651447e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36787 } } peers { permanent_uuid: "522121848ad548a49cf2f2ee25898ec3" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 39019 } }
I20260808 22:22:24.939916 22135 ts_tablet_manager.cc:1428] T 02ea94ba0e3d4970a9c6c9729b15b785 P 3e22dfd749ff4f02b4c9ac55e651447e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260808 22:22:24.939930 22136 raft_consensus.cc:374] T 02ea94ba0e3d4970a9c6c9729b15b785 P 522121848ad548a49cf2f2ee25898ec3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:22:24.939966 22136 raft_consensus.cc:729] T 02ea94ba0e3d4970a9c6c9729b15b785 P 522121848ad548a49cf2f2ee25898ec3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522121848ad548a49cf2f2ee25898ec3, State: Initialized, Role: FOLLOWER
I20260808 22:22:24.940099 22136 consensus_queue.cc:260] T 02ea94ba0e3d4970a9c6c9729b15b785 P 522121848ad548a49cf2f2ee25898ec3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b49d3ab4f0c4429907d17e534a82add" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44137 } } peers { permanent_uuid: "3e22dfd749ff4f02b4c9ac55e651447e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36787 } } peers { permanent_uuid: "522121848ad548a49cf2f2ee25898ec3" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 39019 } }
I20260808 22:22:24.940320 22136 ts_tablet_manager.cc:1428] T 02ea94ba0e3d4970a9c6c9729b15b785 P 522121848ad548a49cf2f2ee25898ec3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 22:22:24.940662 22140 tablet_bootstrap.cc:492] T 02ea94ba0e3d4970a9c6c9729b15b785 P 7b49d3ab4f0c4429907d17e534a82add: Bootstrap starting.
I20260808 22:22:24.941458 22140 tablet_bootstrap.cc:654] T 02ea94ba0e3d4970a9c6c9729b15b785 P 7b49d3ab4f0c4429907d17e534a82add: Neither blocks nor log segments found. Creating new log.
I20260808 22:22:24.942559 22140 tablet_bootstrap.cc:492] T 02ea94ba0e3d4970a9c6c9729b15b785 P 7b49d3ab4f0c4429907d17e534a82add: No bootstrap required, opened a new log
I20260808 22:22:24.942641 22140 ts_tablet_manager.cc:1397] T 02ea94ba0e3d4970a9c6c9729b15b785 P 7b49d3ab4f0c4429907d17e534a82add: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 22:22:24.943018 22140 raft_consensus.cc:348] T 02ea94ba0e3d4970a9c6c9729b15b785 P 7b49d3ab4f0c4429907d17e534a82add [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b49d3ab4f0c4429907d17e534a82add" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44137 } } peers { permanent_uuid: "3e22dfd749ff4f02b4c9ac55e651447e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36787 } } peers { permanent_uuid: "522121848ad548a49cf2f2ee25898ec3" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 39019 } }
I20260808 22:22:24.943130 22140 raft_consensus.cc:374] T 02ea94ba0e3d4970a9c6c9729b15b785 P 7b49d3ab4f0c4429907d17e534a82add [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:22:24.943161 22140 raft_consensus.cc:729] T 02ea94ba0e3d4970a9c6c9729b15b785 P 7b49d3ab4f0c4429907d17e534a82add [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b49d3ab4f0c4429907d17e534a82add, State: Initialized, Role: FOLLOWER
I20260808 22:22:24.943287 22140 consensus_queue.cc:260] T 02ea94ba0e3d4970a9c6c9729b15b785 P 7b49d3ab4f0c4429907d17e534a82add [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b49d3ab4f0c4429907d17e534a82add" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44137 } } peers { permanent_uuid: "3e22dfd749ff4f02b4c9ac55e651447e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36787 } } peers { permanent_uuid: "522121848ad548a49cf2f2ee25898ec3" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 39019 } }
I20260808 22:22:24.944763 22140 ts_tablet_manager.cc:1428] T 02ea94ba0e3d4970a9c6c9729b15b785 P 7b49d3ab4f0c4429907d17e534a82add: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260808 22:22:24.978986 22144 raft_consensus.cc:482] T 02ea94ba0e3d4970a9c6c9729b15b785 P 522121848ad548a49cf2f2ee25898ec3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 22:22:24.979113 22144 raft_consensus.cc:504] T 02ea94ba0e3d4970a9c6c9729b15b785 P 522121848ad548a49cf2f2ee25898ec3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b49d3ab4f0c4429907d17e534a82add" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44137 } } peers { permanent_uuid: "3e22dfd749ff4f02b4c9ac55e651447e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36787 } } peers { permanent_uuid: "522121848ad548a49cf2f2ee25898ec3" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 39019 } }
I20260808 22:22:24.980311 22144 leader_election.cc:288] T 02ea94ba0e3d4970a9c6c9729b15b785 P 522121848ad548a49cf2f2ee25898ec3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b49d3ab4f0c4429907d17e534a82add (127.9.176.193:44137), 3e22dfd749ff4f02b4c9ac55e651447e (127.9.176.194:36787)
I20260808 22:22:24.985455 22127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:24.981111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.195:48303 (local address 127.9.176.193:44137)
0808 22:22:24.981248 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:24.981253 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:24.981270 (+ 17us) server_negotiation.cc:407] Connection header received
0808 22:22:24.981401 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:24.981405 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:24.981452 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:24.981495 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:22:24.981874 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:24.982602 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:24.984146 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:24.984753 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:24.984836 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:24.984908 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:24.985255 (+ 347us) server_negotiation.cc:299] Negotiation successful
0808 22:22:24.985335 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 22:22:24.985857 22253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:24.980564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.193:44137 (local address 127.9.176.195:48303)
0808 22:22:24.981152 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:24.981166 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:24.981260 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:24.981620 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:24.981623 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:24.981640 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:22:24.981763 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:24.981768 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:24.982750 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:24.982753 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:24.984009 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:24.984021 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:24.985015 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:24.985018 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:24.985120 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:24.985146 (+ 26us) client_negotiation.cc:769] Sending connection context
0808 22:22:24.985628 (+ 482us) client_negotiation.cc:240] Negotiation successful
0808 22:22:24.985716 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":94,"threads_started":1}
I20260808 22:22:24.986277 21556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02ea94ba0e3d4970a9c6c9729b15b785" candidate_uuid: "522121848ad548a49cf2f2ee25898ec3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b49d3ab4f0c4429907d17e534a82add" is_pre_election: true
I20260808 22:22:24.986320 22131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:24.980920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.195:41347 (local address 127.9.176.194:36787)
0808 22:22:24.981026 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:24.981031 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:24.981541 (+ 510us) server_negotiation.cc:407] Connection header received
0808 22:22:24.981737 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:24.981741 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:24.981790 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:24.981839 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:22:24.982295 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:24.983034 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:24.984562 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:24.985224 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:24.985322 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:24.985375 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:24.986124 (+ 749us) server_negotiation.cc:299] Negotiation successful
0808 22:22:24.986205 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260808 22:22:24.986415 21556 raft_consensus.cc:2454] T 02ea94ba0e3d4970a9c6c9729b15b785 P 7b49d3ab4f0c4429907d17e534a82add [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 522121848ad548a49cf2f2ee25898ec3 in term 0.
I20260808 22:22:24.986742 21892 leader_election.cc:302] T 02ea94ba0e3d4970a9c6c9729b15b785 P 522121848ad548a49cf2f2ee25898ec3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 522121848ad548a49cf2f2ee25898ec3, 7b49d3ab4f0c4429907d17e534a82add; no voters:
I20260808 22:22:24.986958 22252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:24.980615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.194:36787 (local address 127.9.176.195:41347)
0808 22:22:24.981456 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:24.981470 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:24.981615 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:24.981998 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:24.982001 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:24.982019 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:22:24.982135 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:24.982143 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:24.983211 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:24.983214 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:24.984405 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:24.984417 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:24.985478 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:24.985482 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:24.985774 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:24.985800 (+ 26us) client_negotiation.cc:769] Sending connection context
0808 22:22:24.986669 (+ 869us) client_negotiation.cc:240] Negotiation successful
0808 22:22:24.986747 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":775,"thread_start_us":67,"threads_started":1}
I20260808 22:22:24.986994 22144 raft_consensus.cc:2781] T 02ea94ba0e3d4970a9c6c9729b15b785 P 522121848ad548a49cf2f2ee25898ec3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 22:22:24.987195 22144 raft_consensus.cc:482] T 02ea94ba0e3d4970a9c6c9729b15b785 P 522121848ad548a49cf2f2ee25898ec3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 22:22:24.987239 22144 raft_consensus.cc:3037] T 02ea94ba0e3d4970a9c6c9729b15b785 P 522121848ad548a49cf2f2ee25898ec3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:22:24.987507 21772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02ea94ba0e3d4970a9c6c9729b15b785" candidate_uuid: "522121848ad548a49cf2f2ee25898ec3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e22dfd749ff4f02b4c9ac55e651447e" is_pre_election: true
I20260808 22:22:24.987636 21772 raft_consensus.cc:2454] T 02ea94ba0e3d4970a9c6c9729b15b785 P 3e22dfd749ff4f02b4c9ac55e651447e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 522121848ad548a49cf2f2ee25898ec3 in term 0.
I20260808 22:22:24.988389 22144 raft_consensus.cc:504] T 02ea94ba0e3d4970a9c6c9729b15b785 P 522121848ad548a49cf2f2ee25898ec3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b49d3ab4f0c4429907d17e534a82add" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44137 } } peers { permanent_uuid: "3e22dfd749ff4f02b4c9ac55e651447e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36787 } } peers { permanent_uuid: "522121848ad548a49cf2f2ee25898ec3" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 39019 } }
I20260808 22:22:24.988714 22144 leader_election.cc:288] T 02ea94ba0e3d4970a9c6c9729b15b785 P 522121848ad548a49cf2f2ee25898ec3 [CANDIDATE]: Term 1 election: Requested vote from peers 7b49d3ab4f0c4429907d17e534a82add (127.9.176.193:44137), 3e22dfd749ff4f02b4c9ac55e651447e (127.9.176.194:36787)
I20260808 22:22:24.989109 21772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02ea94ba0e3d4970a9c6c9729b15b785" candidate_uuid: "522121848ad548a49cf2f2ee25898ec3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e22dfd749ff4f02b4c9ac55e651447e"
I20260808 22:22:24.989210 21772 raft_consensus.cc:3037] T 02ea94ba0e3d4970a9c6c9729b15b785 P 3e22dfd749ff4f02b4c9ac55e651447e [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:22:24.990056 21556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02ea94ba0e3d4970a9c6c9729b15b785" candidate_uuid: "522121848ad548a49cf2f2ee25898ec3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b49d3ab4f0c4429907d17e534a82add"
I20260808 22:22:24.990176 21556 raft_consensus.cc:3037] T 02ea94ba0e3d4970a9c6c9729b15b785 P 7b49d3ab4f0c4429907d17e534a82add [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:22:24.990191 21772 raft_consensus.cc:2454] T 02ea94ba0e3d4970a9c6c9729b15b785 P 3e22dfd749ff4f02b4c9ac55e651447e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 522121848ad548a49cf2f2ee25898ec3 in term 1.
I20260808 22:22:24.990479 21889 leader_election.cc:302] T 02ea94ba0e3d4970a9c6c9729b15b785 P 522121848ad548a49cf2f2ee25898ec3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e22dfd749ff4f02b4c9ac55e651447e, 522121848ad548a49cf2f2ee25898ec3; no voters:
I20260808 22:22:24.990620 22144 raft_consensus.cc:2781] T 02ea94ba0e3d4970a9c6c9729b15b785 P 522121848ad548a49cf2f2ee25898ec3 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:22:24.990782 22144 raft_consensus.cc:686] T 02ea94ba0e3d4970a9c6c9729b15b785 P 522121848ad548a49cf2f2ee25898ec3 [term 1 LEADER]: Becoming Leader. State: Replica: 522121848ad548a49cf2f2ee25898ec3, State: Running, Role: LEADER
I20260808 22:22:24.991006 22144 consensus_queue.cc:237] T 02ea94ba0e3d4970a9c6c9729b15b785 P 522121848ad548a49cf2f2ee25898ec3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b49d3ab4f0c4429907d17e534a82add" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44137 } } peers { permanent_uuid: "3e22dfd749ff4f02b4c9ac55e651447e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36787 } } peers { permanent_uuid: "522121848ad548a49cf2f2ee25898ec3" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 39019 } }
I20260808 22:22:24.991464 21556 raft_consensus.cc:2454] T 02ea94ba0e3d4970a9c6c9729b15b785 P 7b49d3ab4f0c4429907d17e534a82add [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 522121848ad548a49cf2f2ee25898ec3 in term 1.
I20260808 22:22:24.993183 21288 catalog_manager.cc:5495] T 02ea94ba0e3d4970a9c6c9729b15b785 P 522121848ad548a49cf2f2ee25898ec3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 522121848ad548a49cf2f2ee25898ec3 (127.9.176.195). New cstate: current_term: 1 leader_uuid: "522121848ad548a49cf2f2ee25898ec3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b49d3ab4f0c4429907d17e534a82add" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44137 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e22dfd749ff4f02b4c9ac55e651447e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "522121848ad548a49cf2f2ee25898ec3" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 39019 } health_report { overall_health: HEALTHY } } }
I20260808 22:22:25.015334 22132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:25.011608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41808 (local address 127.9.176.195:39019)
0808 22:22:25.011718 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:25.011722 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:25.011737 (+ 15us) server_negotiation.cc:407] Connection header received
0808 22:22:25.011894 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:25.011898 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:25.011964 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:25.012077 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:22:25.012250 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:25.013032 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:25.013796 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:25.014137 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:25.014235 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:25.014403 (+ 168us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:22:25.015009 (+ 606us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:22:25.015121 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:25.015151 (+ 30us) server_negotiation.cc:299] Negotiation successful
0808 22:22:25.015203 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260808 22:22:25.025846 22266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:25.022351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.194:36787 (local address 127.9.176.195:37067)
0808 22:22:25.022617 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:25.022628 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:25.022718 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:25.022941 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:25.022943 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:25.022950 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 22:22:25.023044 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:25.023048 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:25.023951 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:25.023954 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:25.024548 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:25.024556 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:25.025131 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:25.025135 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:25.025217 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:25.025263 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 22:22:25.025568 (+ 305us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 22:22:25.025575 (+ 7us) client_negotiation.cc:769] Sending connection context
0808 22:22:25.025640 (+ 65us) client_negotiation.cc:240] Negotiation successful
0808 22:22:25.025719 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":93,"threads_started":1}
I20260808 22:22:25.025841 22131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:25.022485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.195:37067 (local address 127.9.176.194:36787)
0808 22:22:25.022599 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:25.022603 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:25.022687 (+ 84us) server_negotiation.cc:407] Connection header received
0808 22:22:25.022798 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:25.022801 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:25.022849 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:25.022914 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:22:25.023117 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:25.023865 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:25.024644 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:25.025035 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:25.025167 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:25.025375 (+ 208us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:22:25.025479 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:22:25.025542 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:25.025656 (+ 114us) server_negotiation.cc:299] Negotiation successful
0808 22:22:25.025707 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260808 22:22:25.031279 21772 raft_consensus.cc:1261] T 02ea94ba0e3d4970a9c6c9729b15b785 P 3e22dfd749ff4f02b4c9ac55e651447e [term 1 FOLLOWER]: Refusing update from remote peer 522121848ad548a49cf2f2ee25898ec3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:22:25.031353 21556 raft_consensus.cc:1261] T 02ea94ba0e3d4970a9c6c9729b15b785 P 7b49d3ab4f0c4429907d17e534a82add [term 1 FOLLOWER]: Refusing update from remote peer 522121848ad548a49cf2f2ee25898ec3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:22:25.031749 22144 consensus_queue.cc:1035] T 02ea94ba0e3d4970a9c6c9729b15b785 P 522121848ad548a49cf2f2ee25898ec3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e22dfd749ff4f02b4c9ac55e651447e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 36787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:22:25.032102 22144 consensus_queue.cc:1035] T 02ea94ba0e3d4970a9c6c9729b15b785 P 522121848ad548a49cf2f2ee25898ec3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b49d3ab4f0c4429907d17e534a82add" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:22:25.047868 22132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:25.044978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.194:49951 (local address 127.9.176.195:39019)
0808 22:22:25.045088 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:25.045091 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:25.045104 (+ 13us) server_negotiation.cc:407] Connection header received
0808 22:22:25.045205 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:25.045207 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:25.045244 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:25.045336 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:22:25.045534 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:25.046042 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:25.046841 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:25.047137 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:25.047236 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:25.047437 (+ 201us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:22:25.047513 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:22:25.047574 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:25.047700 (+ 126us) server_negotiation.cc:299] Negotiation successful
0808 22:22:25.047747 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260808 22:22:25.047899 22208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:25.044939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.195:39019 (local address 127.9.176.194:49951)
0808 22:22:25.045056 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:25.045070 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:25.045137 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:25.045345 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:25.045347 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:25.045353 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 22:22:25.045454 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:25.045460 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:25.046126 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:25.046128 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:25.046744 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:25.046754 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:25.047231 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:25.047234 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:25.047300 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:25.047327 (+ 27us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 22:22:25.047614 (+ 287us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 22:22:25.047621 (+ 7us) client_negotiation.cc:769] Sending connection context
0808 22:22:25.047684 (+ 63us) client_negotiation.cc:240] Negotiation successful
0808 22:22:25.047772 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260808 22:22:25.070454 21287 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58556:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:22:25.070601 21287 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58556:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:22:25.072438 21287 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 824baa547e754adfbe38c0b466a74c12: Sending DeleteTablet for 3 replicas of tablet 02ea94ba0e3d4970a9c6c9729b15b785
I20260808 22:22:25.072992 21528 tablet_service.cc:1514] Processing DeleteTablet for tablet 02ea94ba0e3d4970a9c6c9729b15b785 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:22:25 UTC) from {username='oryx', principal='oryx/127.9.176.254@KRBTEST.COM'} at 127.0.0.1:45358
I20260808 22:22:25.073004 21743 tablet_service.cc:1514] Processing DeleteTablet for tablet 02ea94ba0e3d4970a9c6c9729b15b785 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:22:25 UTC) from {username='oryx', principal='oryx/127.9.176.254@KRBTEST.COM'} at 127.0.0.1:48144
I20260808 22:22:25.073179 22172 tablet_replica.cc:330] T 02ea94ba0e3d4970a9c6c9729b15b785 P 7b49d3ab4f0c4429907d17e534a82add: stopping tablet replica
I20260808 22:22:25.073292 22172 raft_consensus.cc:2229] T 02ea94ba0e3d4970a9c6c9729b15b785 P 7b49d3ab4f0c4429907d17e534a82add [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:22:25.073364 21968 tablet_service.cc:1514] Processing DeleteTablet for tablet 02ea94ba0e3d4970a9c6c9729b15b785 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:22:25 UTC) from {username='oryx', principal='oryx/127.9.176.254@KRBTEST.COM'} at 127.0.0.1:41794
I20260808 22:22:25.073405 22172 raft_consensus.cc:2258] T 02ea94ba0e3d4970a9c6c9729b15b785 P 7b49d3ab4f0c4429907d17e534a82add [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:22:25.073400 22170 tablet_replica.cc:330] T 02ea94ba0e3d4970a9c6c9729b15b785 P 3e22dfd749ff4f02b4c9ac55e651447e: stopping tablet replica
I20260808 22:22:25.073508 22175 tablet_replica.cc:330] T 02ea94ba0e3d4970a9c6c9729b15b785 P 522121848ad548a49cf2f2ee25898ec3: stopping tablet replica
I20260808 22:22:25.073529 22170 raft_consensus.cc:2229] T 02ea94ba0e3d4970a9c6c9729b15b785 P 3e22dfd749ff4f02b4c9ac55e651447e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:22:25.073594 22175 raft_consensus.cc:2229] T 02ea94ba0e3d4970a9c6c9729b15b785 P 522121848ad548a49cf2f2ee25898ec3 [term 1 LEADER]: Raft consensus shutting down.
I20260808 22:22:25.073647 22170 raft_consensus.cc:2258] T 02ea94ba0e3d4970a9c6c9729b15b785 P 3e22dfd749ff4f02b4c9ac55e651447e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:22:25.073858 22175 raft_consensus.cc:2258] T 02ea94ba0e3d4970a9c6c9729b15b785 P 522121848ad548a49cf2f2ee25898ec3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:22:25.074098 22172 ts_tablet_manager.cc:1905] T 02ea94ba0e3d4970a9c6c9729b15b785 P 7b49d3ab4f0c4429907d17e534a82add: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:22:25.074559 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 21414
I20260808 22:22:25.074667 22175 ts_tablet_manager.cc:1905] T 02ea94ba0e3d4970a9c6c9729b15b785 P 522121848ad548a49cf2f2ee25898ec3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:22:25.076808 22175 ts_tablet_manager.cc:1918] T 02ea94ba0e3d4970a9c6c9729b15b785 P 522121848ad548a49cf2f2ee25898ec3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 22:22:25.076874 22175 log.cc:1196] T 02ea94ba0e3d4970a9c6c9729b15b785 P 522121848ad548a49cf2f2ee25898ec3: Deleting WAL directory at /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/minicluster-data/ts-2/wal/wals/02ea94ba0e3d4970a9c6c9729b15b785
I20260808 22:22:25.077658 22175 ts_tablet_manager.cc:1939] T 02ea94ba0e3d4970a9c6c9729b15b785 P 522121848ad548a49cf2f2ee25898ec3: Deleting consensus metadata
I20260808 22:22:25.078423 21257 catalog_manager.cc:4841] TS 522121848ad548a49cf2f2ee25898ec3 (127.9.176.195:39019): tablet 02ea94ba0e3d4970a9c6c9729b15b785 (table test-table [id=24e3bddce6994c9cbe7bf49b64f6cb67]) successfully deleted
I20260808 22:22:25.085938 22170 ts_tablet_manager.cc:1905] T 02ea94ba0e3d4970a9c6c9729b15b785 P 3e22dfd749ff4f02b4c9ac55e651447e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:22:25.087864 22170 ts_tablet_manager.cc:1918] T 02ea94ba0e3d4970a9c6c9729b15b785 P 3e22dfd749ff4f02b4c9ac55e651447e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 22:22:25.087942 22170 log.cc:1196] T 02ea94ba0e3d4970a9c6c9729b15b785 P 3e22dfd749ff4f02b4c9ac55e651447e: Deleting WAL directory at /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786227706282457-9923-0/minicluster-data/ts-1/wal/wals/02ea94ba0e3d4970a9c6c9729b15b785
I20260808 22:22:25.088169 22170 ts_tablet_manager.cc:1939] T 02ea94ba0e3d4970a9c6c9729b15b785 P 3e22dfd749ff4f02b4c9ac55e651447e: Deleting consensus metadata
I20260808 22:22:25.088796 21256 catalog_manager.cc:4841] TS 3e22dfd749ff4f02b4c9ac55e651447e (127.9.176.194:36787): tablet 02ea94ba0e3d4970a9c6c9729b15b785 (table test-table [id=24e3bddce6994c9cbe7bf49b64f6cb67]) successfully deleted
W20260808 22:22:25.090240 21258 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.176.193:44137 (error 108)
I20260808 22:22:25.090391 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 21636
I20260808 22:22:25.093654 22275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:25.093116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.193:44137 (local address 127.0.0.1:45368)
0808 22:22:25.093441 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:25.093516 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.193:44137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":64,"threads_started":1}
W20260808 22:22:25.093773 21258 catalog_manager.cc:4584] TS 7b49d3ab4f0c4429907d17e534a82add (127.9.176.193:44137): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 02ea94ba0e3d4970a9c6c9729b15b785: Network error: Client connection negotiation failed: client connection to 127.9.176.193:44137: connect: Connection refused (error 111)
I20260808 22:22:25.104758 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 21865
I20260808 22:22:25.115172 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 21233
2026-08-08T22:22:25Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3052 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786227706282457-9923-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 22:22:25 dist-test-slave-ncp9 krb5kdc[22280](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 22:22:25 dist-test-slave-ncp9 krb5kdc[22280](info): set up 2 sockets
Aug 08 22:22:25 dist-test-slave-ncp9 krb5kdc[22280](info): commencing operation
krb5kdc: starting...
W20260808 22:22:26.240988 9923 mini_kdc.cc:121] Time spent starting KDC: real 1.099s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786227706282457-9923-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786227706282457-9923-0/krb5kdc/test-user.keytab.
Aug 08 22:22:26 dist-test-slave-ncp9 krb5kdc[22280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227746, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T22:22:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:22:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.176.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.176.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.176.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab.
Entry for principal kudu/127.9.176.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab.
I20260808 22:22:26.355644 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786227706282457-9923-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786227706282457-9923-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786227706282457-9923-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:41783
--webserver_interface=127.9.176.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab
--principal=kudu/127.9.176.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:41773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.176.254:41783
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:22:26.462208 22299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:22:26.462450 22299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:22:26.462502 22299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:22:26.466367 22299 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:22:26.466418 22299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:22:26.466442 22299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:22:26.466462 22299 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:22:26.466482 22299 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:22:26.471479 22299 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:41773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786227706282457-9923-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786227706282457-9923-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.176.254:41783
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:41783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.176.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22299
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786227706282457-9923-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:22:26.472402 22299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:22:26.473408 22299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:22:26.481475 22311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:26.481460 22309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:26.481462 22313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:22:26.481802 22299 server_base.cc:1034] running on GCE node
I20260808 22:22:26.482321 22299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:22:26.483286 22299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:22:26.484438 22299 hybrid_clock.cc:648] HybridClock initialized: now 1786227746484415 us; error 38 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-08T22:22:26Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1372 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 22:22:26 dist-test-slave-ncp9 krb5kdc[22322](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 22:22:26 dist-test-slave-ncp9 krb5kdc[22322](info): set up 2 sockets
Aug 08 22:22:26 dist-test-slave-ncp9 krb5kdc[22322](info): commencing operation
krb5kdc: starting...
W20260808 22:22:27.614319 9923 mini_kdc.cc:121] Time spent starting KDC: real 1.100s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/krb5kdc/test-user.keytab.
Aug 08 22:22:27 dist-test-slave-ncp9 krb5kdc[22322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T22:22:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:22:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.176.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.176.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.176.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab.
Entry for principal kudu/127.9.176.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab.
I20260808 22:22:27.732273 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:44973
--webserver_interface=127.9.176.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab
--principal=kudu/127.9.176.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:40251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.176.254:44973
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:22:27.840323 22343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:22:27.840551 22343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:22:27.840587 22343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:22:27.844427 22343 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:22:27.844475 22343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:22:27.844492 22343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:22:27.844511 22343 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:22:27.844528 22343 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:22:27.849457 22343 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:40251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.176.254:44973
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab
--principal=kudu/127.9.176.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:44973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.176.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22343
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:22:27.850325 22343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:22:27.851244 22343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:22:27.859292 22354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:27.859280 22356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:22:27.859369 22343 server_base.cc:1034] running on GCE node
W20260808 22:22:27.859280 22359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:22:27.860044 22343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:22:27.861068 22343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:22:27.862224 22343 hybrid_clock.cc:648] HybridClock initialized: now 1786227747862204 us; error 31 us; skew 500 ppm
Aug 08 22:22:27 dist-test-slave-ncp9 krb5kdc[22322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227747, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:22:27.867517 22343 init.cc:377] Logged in from keytab as kudu/127.9.176.254@KRBTEST.COM (short username kudu)
I20260808 22:22:27.868363 22343 webserver.cc:460] Webserver started at http://127.9.176.254:45155/ using document root <none> and password file <none>
I20260808 22:22:27.868885 22343 fs_manager.cc:359] Metadata directory not provided
I20260808 22:22:27.868992 22343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:22:27.869206 22343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:22:27.871173 22343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/master-0/data/instance:
uuid: "5c245739a3c24acc94a68c4f6dda2724"
format_stamp: "Formatted at 2026-08-08 22:22:27 on dist-test-slave-ncp9"
server_key: "77adce4c5421ec52fa84a5bcf77fd931"
server_key_iv: "36ad22acd4a07cd9be43f57ae8f82eb6"
server_key_version: "encryptionkey@0"
I20260808 22:22:27.871711 22343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/master-0/wal/instance:
uuid: "5c245739a3c24acc94a68c4f6dda2724"
format_stamp: "Formatted at 2026-08-08 22:22:27 on dist-test-slave-ncp9"
server_key: "77adce4c5421ec52fa84a5bcf77fd931"
server_key_iv: "36ad22acd4a07cd9be43f57ae8f82eb6"
server_key_version: "encryptionkey@0"
I20260808 22:22:27.875237 22343 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 22:22:27.877610 22372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:27.878767 22343 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 22:22:27.878880 22343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/master-0/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/master-0/wal
uuid: "5c245739a3c24acc94a68c4f6dda2724"
format_stamp: "Formatted at 2026-08-08 22:22:27 on dist-test-slave-ncp9"
server_key: "77adce4c5421ec52fa84a5bcf77fd931"
server_key_iv: "36ad22acd4a07cd9be43f57ae8f82eb6"
server_key_version: "encryptionkey@0"
I20260808 22:22:27.878969 22343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:22:27.899602 22343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:22:27.902667 22343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:22:27.902843 22343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:22:27.909422 22343 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.254:44973
I20260808 22:22:27.909446 22449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.254:44973 every 8 connection(s)
I20260808 22:22:27.910328 22343 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
I20260808 22:22:27.912962 22451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:22:27.917693 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 22343
I20260808 22:22:27.917814 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/master-0/wal/instance
I20260808 22:22:27.918129 9923 external_mini_cluster.cc:1428] Setting key 5d87e4667e0bc678d0ae8f96dd55f31b
I20260808 22:22:27.918617 22451 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c245739a3c24acc94a68c4f6dda2724: Bootstrap starting.
I20260808 22:22:27.920876 22451 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c245739a3c24acc94a68c4f6dda2724: Neither blocks nor log segments found. Creating new log.
I20260808 22:22:27.921609 22451 log.cc:824] T 00000000000000000000000000000000 P 5c245739a3c24acc94a68c4f6dda2724: Log is configured to *not* fsync() on all Append() calls
I20260808 22:22:27.923349 22451 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c245739a3c24acc94a68c4f6dda2724: No bootstrap required, opened a new log
Aug 08 22:22:27 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
I20260808 22:22:27.927055 22451 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c245739a3c24acc94a68c4f6dda2724 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c245739a3c24acc94a68c4f6dda2724" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 44973 } }
I20260808 22:22:27.927268 22451 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5c245739a3c24acc94a68c4f6dda2724 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:22:27.927325 22451 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c245739a3c24acc94a68c4f6dda2724 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c245739a3c24acc94a68c4f6dda2724, State: Initialized, Role: FOLLOWER
I20260808 22:22:27.927737 22451 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c245739a3c24acc94a68c4f6dda2724 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c245739a3c24acc94a68c4f6dda2724" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 44973 } }
I20260808 22:22:27.927857 22451 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5c245739a3c24acc94a68c4f6dda2724 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:22:27.927938 22451 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c245739a3c24acc94a68c4f6dda2724 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:22:27.928056 22451 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c245739a3c24acc94a68c4f6dda2724 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:22:27.928864 22451 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c245739a3c24acc94a68c4f6dda2724 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c245739a3c24acc94a68c4f6dda2724" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 44973 } }
I20260808 22:22:27.929229 22451 leader_election.cc:302] T 00000000000000000000000000000000 P 5c245739a3c24acc94a68c4f6dda2724 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c245739a3c24acc94a68c4f6dda2724; no voters:
I20260808 22:22:27.929572 22451 leader_election.cc:288] T 00000000000000000000000000000000 P 5c245739a3c24acc94a68c4f6dda2724 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:22:27.929643 22461 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c245739a3c24acc94a68c4f6dda2724 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:22:27.929858 22461 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c245739a3c24acc94a68c4f6dda2724 [term 1 LEADER]: Becoming Leader. State: Replica: 5c245739a3c24acc94a68c4f6dda2724, State: Running, Role: LEADER
I20260808 22:22:27.930200 22461 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c245739a3c24acc94a68c4f6dda2724 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c245739a3c24acc94a68c4f6dda2724" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 44973 } }
I20260808 22:22:27.930796 22451 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c245739a3c24acc94a68c4f6dda2724 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:22:27.931793 22463 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c245739a3c24acc94a68c4f6dda2724 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c245739a3c24acc94a68c4f6dda2724. Latest consensus state: current_term: 1 leader_uuid: "5c245739a3c24acc94a68c4f6dda2724" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c245739a3c24acc94a68c4f6dda2724" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 44973 } } }
I20260808 22:22:27.931914 22463 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c245739a3c24acc94a68c4f6dda2724 [sys.catalog]: This master's current role is: LEADER
I20260808 22:22:27.932047 22458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:27.919416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51714 (local address 127.9.176.254:44973)
0808 22:22:27.919800 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:27.919814 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:27.919849 (+ 35us) server_negotiation.cc:407] Connection header received
0808 22:22:27.920462 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:27.920479 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:27.920793 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:27.921133 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:27.921924 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:27.922920 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:27.923814 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:27.924280 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:27.924428 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:27.926992 (+ 2564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:27.926997 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:27.927010 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:22:27.927036 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:27.928672 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:27.929181 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:27.929185 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:27.929189 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:27.929301 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:27.929593 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:27.929595 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:27.929597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:27.930082 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:27.930282 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:27.930848 (+ 566us) server_negotiation.cc:299] Negotiation successful
0808 22:22:27.931105 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":103,"threads_started":1}
I20260808 22:22:27.932219 22462 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c245739a3c24acc94a68c4f6dda2724 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5c245739a3c24acc94a68c4f6dda2724" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c245739a3c24acc94a68c4f6dda2724" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 44973 } } }
I20260808 22:22:27.932294 22462 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c245739a3c24acc94a68c4f6dda2724 [sys.catalog]: This master's current role is: LEADER
I20260808 22:22:27.932298 22472 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:22:27.935302 22472 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:22:27.940078 22472 catalog_manager.cc:1347] Generated new cluster ID: 2fda19fa9dcb4b3f87d21c69c0529ba2
I20260808 22:22:27.940141 22472 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:22:27.948472 22472 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:22:27.949360 22472 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:22:27.961093 22472 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5c245739a3c24acc94a68c4f6dda2724: Generated new TSK 0
I20260808 22:22:27.961676 22472 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.176.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.176.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.176.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.193.keytab.
Entry for principal kudu/127.9.176.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.193.keytab.
I20260808 22:22:27.988633 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.193:0
--local_ip_for_outbound_sockets=127.9.176.193
--webserver_interface=127.9.176.193
--webserver_port=0
--tserver_master_addrs=127.9.176.254:44973
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.193.keytab
--principal=kudu/127.9.176.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:40251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:22:28.096941 22496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:22:28.097169 22496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:22:28.097211 22496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:22:28.101079 22496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:22:28.101128 22496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:22:28.101197 22496 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.193
I20260808 22:22:28.106236 22496 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:40251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.193.keytab
--principal=kudu/127.9.176.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.176.193
--webserver_port=0
--tserver_master_addrs=127.9.176.254:44973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22496
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.193
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:22:28.107188 22496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:22:28.108139 22496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:22:28.117173 22503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:28.117177 22509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:28.117172 22504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:22:28.117388 22496 server_base.cc:1034] running on GCE node
I20260808 22:22:28.117841 22496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:22:28.118417 22496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:22:28.119573 22496 hybrid_clock.cc:648] HybridClock initialized: now 1786227748119559 us; error 26 us; skew 500 ppm
Aug 08 22:22:28 dist-test-slave-ncp9 krb5kdc[22322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227748, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:22:28.122205 22496 init.cc:377] Logged in from keytab as kudu/127.9.176.193@KRBTEST.COM (short username kudu)
I20260808 22:22:28.123013 22496 webserver.cc:460] Webserver started at http://127.9.176.193:37497/ using document root <none> and password file <none>
I20260808 22:22:28.123517 22496 fs_manager.cc:359] Metadata directory not provided
I20260808 22:22:28.123580 22496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:22:28.123775 22496 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:22:28.125756 22496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/ts-0/data/instance:
uuid: "a09819ea0dac4387a5674f5ed473aa19"
format_stamp: "Formatted at 2026-08-08 22:22:28 on dist-test-slave-ncp9"
server_key: "3fe50f25e17755b4c79b07ccce5a6733"
server_key_iv: "5d32c017ed7876341c43415ad94a9e84"
server_key_version: "encryptionkey@0"
I20260808 22:22:28.126272 22496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/ts-0/wal/instance:
uuid: "a09819ea0dac4387a5674f5ed473aa19"
format_stamp: "Formatted at 2026-08-08 22:22:28 on dist-test-slave-ncp9"
server_key: "3fe50f25e17755b4c79b07ccce5a6733"
server_key_iv: "5d32c017ed7876341c43415ad94a9e84"
server_key_version: "encryptionkey@0"
I20260808 22:22:28.129725 22496 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 22:22:28.131922 22530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:28.133184 22496 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 22:22:28.133298 22496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/ts-0/wal
uuid: "a09819ea0dac4387a5674f5ed473aa19"
format_stamp: "Formatted at 2026-08-08 22:22:28 on dist-test-slave-ncp9"
server_key: "3fe50f25e17755b4c79b07ccce5a6733"
server_key_iv: "5d32c017ed7876341c43415ad94a9e84"
server_key_version: "encryptionkey@0"
I20260808 22:22:28.133386 22496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:22:28.145511 22496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:22:28.148336 22496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:22:28.148497 22496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:22:28.149035 22496 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:22:28.149797 22496 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:22:28.149843 22496 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:28.149892 22496 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:22:28.149921 22496 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:28.158352 22496 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.193:33877
I20260808 22:22:28.158375 22702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.193:33877 every 8 connection(s)
I20260808 22:22:28.159225 22496 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
I20260808 22:22:28.164162 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 22496
I20260808 22:22:28.164307 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/ts-0/wal/instance
I20260808 22:22:28.164631 9923 external_mini_cluster.cc:1428] Setting key 15cf250fcb5d7f9eedb12de6e4704d19
Aug 08 22:22:28 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227748, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.193@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
I20260808 22:22:28.172662 22458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:28.160917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.193:46797 (local address 127.9.176.254:44973)
0808 22:22:28.161059 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:28.161064 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:28.162067 (+ 1003us) server_negotiation.cc:407] Connection header received
0808 22:22:28.162895 (+ 828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:28.162898 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:28.162951 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:28.163021 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:28.164236 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:28.164879 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:28.166357 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:28.166589 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:28.166677 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:28.169274 (+ 2597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:28.169277 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:28.169279 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:22:28.169302 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:28.170644 (+ 1342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:28.171207 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:28.171209 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:28.171210 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:28.171277 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:28.171624 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:28.171626 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:28.171627 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:28.171787 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:28.171855 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:28.172448 (+ 593us) server_negotiation.cc:299] Negotiation successful
0808 22:22:28.172538 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260808 22:22:28.173858 22705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:28.161300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:44973 (local address 127.9.176.193:46797)
0808 22:22:28.161918 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:28.161956 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:28.162723 (+ 767us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:28.163170 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:28.163176 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:28.163510 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:28.164090 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:28.164113 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:28.165011 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:28.165015 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:28.166240 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:28.166250 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:28.166733 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:28.166738 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:28.166904 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:28.167435 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:28.167450 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:28.169113 (+ 1663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:22:28.170753 (+ 1640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:28.170759 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:28.170775 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:28.171122 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:28.171355 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:28.171359 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:28.171360 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:28.171544 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:28.171880 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:22:28.171888 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:22:28.172215 (+ 327us) client_negotiation.cc:769] Sending connection context
0808 22:22:28.172432 (+ 217us) client_negotiation.cc:240] Negotiation successful
0808 22:22:28.172797 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":111,"threads_started":1}
WARNING: no policy specified for kudu/127.9.176.194@KRBTEST.COM; defaulting to no policy
I20260808 22:22:28.175057 22703 heartbeater.cc:347] Connected to a master server at 127.9.176.254:44973
I20260808 22:22:28.175302 22703 heartbeater.cc:464] Registering TS with master...
I20260808 22:22:28.175757 22703 heartbeater.cc:511] Master 127.9.176.254:44973 requested a full tablet report, sending...
I20260808 22:22:28.177484 22400 ts_manager.cc:194] Registered new tserver with Master: a09819ea0dac4387a5674f5ed473aa19 (127.9.176.193:33877)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.176.194@KRBTEST.COM" created.
I20260808 22:22:28.178655 22400 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.176.193@KRBTEST.COM'} at 127.9.176.193:46797
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.176.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.194.keytab.
Entry for principal kudu/127.9.176.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.194.keytab.
I20260808 22:22:28.192131 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.194:0
--local_ip_for_outbound_sockets=127.9.176.194
--webserver_interface=127.9.176.194
--webserver_port=0
--tserver_master_addrs=127.9.176.254:44973
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.194.keytab
--principal=kudu/127.9.176.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:40251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:22:28.301527 22711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:22:28.301764 22711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:22:28.301811 22711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:22:28.305716 22711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:22:28.305768 22711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:22:28.305852 22711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.194
I20260808 22:22:28.310811 22711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:40251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.194.keytab
--principal=kudu/127.9.176.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.176.194
--webserver_port=0
--tserver_master_addrs=127.9.176.254:44973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.194
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:22:28.311717 22711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:22:28.312654 22711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:22:28.321895 22727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:28.321940 22720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:22:28.321973 22711 server_base.cc:1034] running on GCE node
W20260808 22:22:28.321908 22724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:22:28.322585 22711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:22:28.323140 22711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:22:28.324307 22711 hybrid_clock.cc:648] HybridClock initialized: now 1786227748324292 us; error 27 us; skew 500 ppm
Aug 08 22:22:28 dist-test-slave-ncp9 krb5kdc[22322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227748, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:22:28.327069 22711 init.cc:377] Logged in from keytab as kudu/127.9.176.194@KRBTEST.COM (short username kudu)
I20260808 22:22:28.327905 22711 webserver.cc:460] Webserver started at http://127.9.176.194:46445/ using document root <none> and password file <none>
I20260808 22:22:28.328408 22711 fs_manager.cc:359] Metadata directory not provided
I20260808 22:22:28.328472 22711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:22:28.328670 22711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:22:28.330693 22711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/ts-1/data/instance:
uuid: "3c4df74acdef4954b44781bc97baa3c2"
format_stamp: "Formatted at 2026-08-08 22:22:28 on dist-test-slave-ncp9"
server_key: "7216e2925575d79aa4af758342b4bb28"
server_key_iv: "f97392a55ab2d8ee4183a476ad5c5de6"
server_key_version: "encryptionkey@0"
I20260808 22:22:28.331228 22711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/ts-1/wal/instance:
uuid: "3c4df74acdef4954b44781bc97baa3c2"
format_stamp: "Formatted at 2026-08-08 22:22:28 on dist-test-slave-ncp9"
server_key: "7216e2925575d79aa4af758342b4bb28"
server_key_iv: "f97392a55ab2d8ee4183a476ad5c5de6"
server_key_version: "encryptionkey@0"
I20260808 22:22:28.334895 22711 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 22:22:28.337265 22737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:28.338411 22711 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 22:22:28.338517 22711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/ts-1/wal
uuid: "3c4df74acdef4954b44781bc97baa3c2"
format_stamp: "Formatted at 2026-08-08 22:22:28 on dist-test-slave-ncp9"
server_key: "7216e2925575d79aa4af758342b4bb28"
server_key_iv: "f97392a55ab2d8ee4183a476ad5c5de6"
server_key_version: "encryptionkey@0"
I20260808 22:22:28.338613 22711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:22:28.352779 22711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:22:28.355425 22711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:22:28.355598 22711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:22:28.356155 22711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:22:28.356976 22711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:22:28.357023 22711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:28.357074 22711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:22:28.357100 22711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:28.365175 22711 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.194:36185
I20260808 22:22:28.365226 22908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.194:36185 every 8 connection(s)
I20260808 22:22:28.366012 22711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
I20260808 22:22:28.367707 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 22711
I20260808 22:22:28.367823 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/ts-1/wal/instance
I20260808 22:22:28.368099 9923 external_mini_cluster.cc:1428] Setting key 583cc8b87f5ffdb08e855fa9689e9102
Aug 08 22:22:28 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227748, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.194@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.176.195@KRBTEST.COM; defaulting to no policy
I20260808 22:22:28.379411 22458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:28.367718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.194:50741 (local address 127.9.176.254:44973)
0808 22:22:28.367870 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:28.367875 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:28.368961 (+ 1086us) server_negotiation.cc:407] Connection header received
0808 22:22:28.369966 (+ 1005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:28.369968 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:28.370010 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:28.370065 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:28.371470 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:28.371984 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:28.373437 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:28.373661 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:28.373748 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:28.376183 (+ 2435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:28.376186 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:28.376188 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:22:28.376211 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:28.377565 (+ 1354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:28.378028 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:28.378032 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:28.378033 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:28.378115 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:28.378446 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:28.378450 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:28.378452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:28.378693 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:28.378779 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:28.379185 (+ 406us) server_negotiation.cc:299] Negotiation successful
0808 22:22:28.379294 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 22:22:28.380182 22913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:28.368103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:44973 (local address 127.9.176.194:50741)
0808 22:22:28.368796 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:28.368849 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:28.369741 (+ 892us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:28.370244 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:28.370252 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:28.370717 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:28.371336 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:28.371355 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:28.372104 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:28.372107 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:28.373339 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:28.373349 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:28.373810 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:28.373814 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:28.373969 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:28.374493 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:28.374509 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:28.376030 (+ 1521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:22:28.377674 (+ 1644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:28.377680 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:28.377694 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:28.377935 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:28.378198 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:28.378201 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:28.378203 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:28.378332 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:28.378782 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:22:28.378787 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:22:28.379000 (+ 213us) client_negotiation.cc:769] Sending connection context
0808 22:22:28.379148 (+ 148us) client_negotiation.cc:240] Negotiation successful
0808 22:22:28.379439 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":119,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.176.195@KRBTEST.COM" created.
I20260808 22:22:28.381325 22909 heartbeater.cc:347] Connected to a master server at 127.9.176.254:44973
I20260808 22:22:28.381546 22909 heartbeater.cc:464] Registering TS with master...
I20260808 22:22:28.381978 22909 heartbeater.cc:511] Master 127.9.176.254:44973 requested a full tablet report, sending...
I20260808 22:22:28.383074 22400 ts_manager.cc:194] Registered new tserver with Master: 3c4df74acdef4954b44781bc97baa3c2 (127.9.176.194:36185)
I20260808 22:22:28.383816 22400 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.176.194@KRBTEST.COM'} at 127.9.176.194:50741
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.176.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.195.keytab.
Entry for principal kudu/127.9.176.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.195.keytab.
I20260808 22:22:28.396100 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.195:0
--local_ip_for_outbound_sockets=127.9.176.195
--webserver_interface=127.9.176.195
--webserver_port=0
--tserver_master_addrs=127.9.176.254:44973
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.195.keytab
--principal=kudu/127.9.176.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:40251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:22:28.503573 22919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:22:28.503823 22919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:22:28.503881 22919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:22:28.507833 22919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:22:28.507891 22919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:22:28.507970 22919 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.195
I20260808 22:22:28.513017 22919 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:40251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.195.keytab
--principal=kudu/127.9.176.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.176.195
--webserver_port=0
--tserver_master_addrs=127.9.176.254:44973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22919
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.195
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:22:28.514002 22919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:22:28.514964 22919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 22:22:28.523943 22919 server_base.cc:1034] running on GCE node
W20260808 22:22:28.523895 22926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:28.523953 22927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:28.523886 22931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:22:28.524535 22919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:22:28.525197 22919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:22:28.526352 22919 hybrid_clock.cc:648] HybridClock initialized: now 1786227748526336 us; error 28 us; skew 500 ppm
Aug 08 22:22:28 dist-test-slave-ncp9 krb5kdc[22322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227748, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:22:28.528939 22919 init.cc:377] Logged in from keytab as kudu/127.9.176.195@KRBTEST.COM (short username kudu)
I20260808 22:22:28.529747 22919 webserver.cc:460] Webserver started at http://127.9.176.195:44475/ using document root <none> and password file <none>
I20260808 22:22:28.530252 22919 fs_manager.cc:359] Metadata directory not provided
I20260808 22:22:28.530316 22919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:22:28.530496 22919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:22:28.532617 22919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/ts-2/data/instance:
uuid: "844363fb7a8f41efb89a2f276c41ba15"
format_stamp: "Formatted at 2026-08-08 22:22:28 on dist-test-slave-ncp9"
server_key: "1c1692659de51d73e5787772867df48f"
server_key_iv: "514b00de909e582070f4c791b4f577ce"
server_key_version: "encryptionkey@0"
I20260808 22:22:28.533183 22919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/ts-2/wal/instance:
uuid: "844363fb7a8f41efb89a2f276c41ba15"
format_stamp: "Formatted at 2026-08-08 22:22:28 on dist-test-slave-ncp9"
server_key: "1c1692659de51d73e5787772867df48f"
server_key_iv: "514b00de909e582070f4c791b4f577ce"
server_key_version: "encryptionkey@0"
I20260808 22:22:28.536705 22919 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260808 22:22:28.539033 22944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:28.540198 22919 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 22:22:28.540309 22919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/ts-2/wal
uuid: "844363fb7a8f41efb89a2f276c41ba15"
format_stamp: "Formatted at 2026-08-08 22:22:28 on dist-test-slave-ncp9"
server_key: "1c1692659de51d73e5787772867df48f"
server_key_iv: "514b00de909e582070f4c791b4f577ce"
server_key_version: "encryptionkey@0"
I20260808 22:22:28.540398 22919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:22:28.552681 22919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:22:28.555210 22919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:22:28.555375 22919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:22:28.555908 22919 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:22:28.556668 22919 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:22:28.556716 22919 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:28.556766 22919 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:22:28.556793 22919 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:28.564950 22919 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.195:42469
I20260808 22:22:28.565004 23122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.195:42469 every 8 connection(s)
I20260808 22:22:28.565778 22919 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
I20260808 22:22:28.571476 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 22919
I20260808 22:22:28.571575 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/ts-2/wal/instance
I20260808 22:22:28.571825 9923 external_mini_cluster.cc:1428] Setting key 363cb84fb7cf3759cf525d58ac57dea5
Aug 08 22:22:28 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227748, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.195@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
I20260808 22:22:28.577966 22458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:28.567400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.195:54239 (local address 127.9.176.254:44973)
0808 22:22:28.567519 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:28.567522 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:28.568491 (+ 969us) server_negotiation.cc:407] Connection header received
0808 22:22:28.569405 (+ 914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:28.569408 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:28.569468 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:28.569543 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:28.570650 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:28.571196 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:28.572398 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:28.572615 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:28.572713 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:28.574934 (+ 2221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:28.574938 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:28.574940 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:22:28.574964 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:28.576066 (+ 1102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:28.576526 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:28.576530 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:28.576531 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:28.576612 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:28.576938 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:28.576941 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:28.576943 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:28.577118 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:28.577205 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:28.577689 (+ 484us) server_negotiation.cc:299] Negotiation successful
0808 22:22:28.577810 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260808 22:22:28.578640 23128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:28.567722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:44973 (local address 127.9.176.195:54239)
0808 22:22:28.568352 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:28.568392 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:28.569203 (+ 811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:28.569718 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:28.569725 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:28.570051 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:28.570503 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:28.570516 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:28.571316 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:28.571318 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:28.572271 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:28.572282 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:28.572746 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:28.572750 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:28.572945 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:28.573443 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:28.573458 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:28.574784 (+ 1326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:22:28.576178 (+ 1394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:28.576184 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:28.576197 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:28.576432 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:28.576694 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:28.576698 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:28.576700 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:28.576837 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:28.577227 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:22:28.577233 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:22:28.577454 (+ 221us) client_negotiation.cc:769] Sending connection context
0808 22:22:28.577601 (+ 147us) client_negotiation.cc:240] Negotiation successful
0808 22:22:28.577895 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":114,"threads_started":1}
I20260808 22:22:28.579595 23124 heartbeater.cc:347] Connected to a master server at 127.9.176.254:44973
I20260808 22:22:28.579789 23124 heartbeater.cc:464] Registering TS with master...
I20260808 22:22:28.580199 23124 heartbeater.cc:511] Master 127.9.176.254:44973 requested a full tablet report, sending...
I20260808 22:22:28.581105 22400 ts_manager.cc:194] Registered new tserver with Master: 844363fb7a8f41efb89a2f276c41ba15 (127.9.176.195:42469)
I20260808 22:22:28.581727 22400 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.176.195@KRBTEST.COM'} at 127.9.176.195:54239
I20260808 22:22:28.585175 9923 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.9.176.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.176.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.176.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/krb5kdc/oryx_127.9.176.193.keytab.
Entry for principal oryx/127.9.176.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/krb5kdc/oryx_127.9.176.193.keytab.
WARNING: no policy specified for oryx/127.9.176.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.176.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.176.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/krb5kdc/oryx_127.9.176.194.keytab.
Entry for principal oryx/127.9.176.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/krb5kdc/oryx_127.9.176.194.keytab.
WARNING: no policy specified for oryx/127.9.176.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.176.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.176.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/krb5kdc/oryx_127.9.176.195.keytab.
Entry for principal oryx/127.9.176.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/krb5kdc/oryx_127.9.176.195.keytab.
I20260808 22:22:28.651557 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 22496
I20260808 22:22:28.666224 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 22711
I20260808 22:22:28.680078 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 22919
I20260808 22:22:28.689502 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 22343
I20260808 22:22:28.701246 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:44973
--webserver_interface=127.9.176.254
--webserver_port=45155
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab
--principal=kudu/127.9.176.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:40251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.176.254:44973
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:22:28.808377 23141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:22:28.808607 23141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:22:28.808661 23141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:22:28.812506 23141 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:22:28.812557 23141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:22:28.812584 23141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:22:28.812605 23141 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:22:28.812633 23141 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:22:28.817623 23141 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:40251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.176.254:44973
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab
--principal=kudu/127.9.176.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:44973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.176.254
--webserver_port=45155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23141
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:22:28.818506 23141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:22:28.819391 23141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:22:28.827107 23148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:28.827116 23147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:28.827128 23152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:22:28.827389 23141 server_base.cc:1034] running on GCE node
I20260808 22:22:28.827906 23141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:22:28.828876 23141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:22:28.830061 23141 hybrid_clock.cc:648] HybridClock initialized: now 1786227748830044 us; error 28 us; skew 500 ppm
Aug 08 22:22:28 dist-test-slave-ncp9 krb5kdc[22322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227748, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:22:28.832612 23141 init.cc:377] Logged in from keytab as kudu/127.9.176.254@KRBTEST.COM (short username kudu)
I20260808 22:22:28.833469 23141 webserver.cc:460] Webserver started at http://127.9.176.254:45155/ using document root <none> and password file <none>
I20260808 22:22:28.833935 23141 fs_manager.cc:359] Metadata directory not provided
I20260808 22:22:28.834002 23141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:22:28.837545 23141 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260808 22:22:28.839488 23162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:28.840613 23141 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 22:22:28.840716 23141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/master-0/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/master-0/wal
uuid: "5c245739a3c24acc94a68c4f6dda2724"
format_stamp: "Formatted at 2026-08-08 22:22:27 on dist-test-slave-ncp9"
server_key: "77adce4c5421ec52fa84a5bcf77fd931"
server_key_iv: "36ad22acd4a07cd9be43f57ae8f82eb6"
server_key_version: "encryptionkey@0"
I20260808 22:22:28.841267 23141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:22:28.855973 23141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:22:28.858487 23141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:22:28.858644 23141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:22:28.865221 23141 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.254:44973
I20260808 22:22:28.865235 23249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.254:44973 every 8 connection(s)
I20260808 22:22:28.866187 23141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
I20260808 22:22:28.867295 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 23141
I20260808 22:22:28.867704 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.193:33877
--local_ip_for_outbound_sockets=127.9.176.193
--tserver_master_addrs=127.9.176.254:44973
--webserver_port=37497
--webserver_interface=127.9.176.193
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.193.keytab
--principal=kudu/127.9.176.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:40251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.9.176.193
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/krb5kdc/oryx_127.9.176.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 22:22:28.870223 23251 sys_catalog.cc:263] Verifying existing consensus state
I20260808 22:22:28.871771 23251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c245739a3c24acc94a68c4f6dda2724: Bootstrap starting.
I20260808 22:22:28.887632 23251 log.cc:824] T 00000000000000000000000000000000 P 5c245739a3c24acc94a68c4f6dda2724: Log is configured to *not* fsync() on all Append() calls
I20260808 22:22:28.894407 23251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c245739a3c24acc94a68c4f6dda2724: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 22:22:28.894749 23251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c245739a3c24acc94a68c4f6dda2724: Bootstrap complete.
I20260808 22:22:28.898753 23251 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c245739a3c24acc94a68c4f6dda2724 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c245739a3c24acc94a68c4f6dda2724" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 44973 } }
I20260808 22:22:28.899147 23251 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c245739a3c24acc94a68c4f6dda2724 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c245739a3c24acc94a68c4f6dda2724, State: Initialized, Role: FOLLOWER
I20260808 22:22:28.899600 23251 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c245739a3c24acc94a68c4f6dda2724 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c245739a3c24acc94a68c4f6dda2724" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 44973 } }
I20260808 22:22:28.899717 23251 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5c245739a3c24acc94a68c4f6dda2724 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:22:28.899785 23251 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c245739a3c24acc94a68c4f6dda2724 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:22:28.899878 23251 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c245739a3c24acc94a68c4f6dda2724 [term 1 FOLLOWER]: Advancing to term 2
I20260808 22:22:28.901732 23251 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c245739a3c24acc94a68c4f6dda2724 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c245739a3c24acc94a68c4f6dda2724" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 44973 } }
I20260808 22:22:28.902029 23251 leader_election.cc:302] T 00000000000000000000000000000000 P 5c245739a3c24acc94a68c4f6dda2724 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c245739a3c24acc94a68c4f6dda2724; no voters:
I20260808 22:22:28.902326 23251 leader_election.cc:288] T 00000000000000000000000000000000 P 5c245739a3c24acc94a68c4f6dda2724 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260808 22:22:28.902426 23258 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c245739a3c24acc94a68c4f6dda2724 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 22:22:28.902664 23258 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c245739a3c24acc94a68c4f6dda2724 [term 2 LEADER]: Becoming Leader. State: Replica: 5c245739a3c24acc94a68c4f6dda2724, State: Running, Role: LEADER
I20260808 22:22:28.903009 23258 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c245739a3c24acc94a68c4f6dda2724 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c245739a3c24acc94a68c4f6dda2724" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 44973 } }
I20260808 22:22:28.903718 23251 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c245739a3c24acc94a68c4f6dda2724 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:22:28.904070 23258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c245739a3c24acc94a68c4f6dda2724 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c245739a3c24acc94a68c4f6dda2724. Latest consensus state: current_term: 2 leader_uuid: "5c245739a3c24acc94a68c4f6dda2724" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c245739a3c24acc94a68c4f6dda2724" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 44973 } } }
I20260808 22:22:28.904196 23258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c245739a3c24acc94a68c4f6dda2724 [sys.catalog]: This master's current role is: LEADER
I20260808 22:22:28.904539 23259 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c245739a3c24acc94a68c4f6dda2724 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5c245739a3c24acc94a68c4f6dda2724" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c245739a3c24acc94a68c4f6dda2724" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 44973 } } }
I20260808 22:22:28.904714 23259 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c245739a3c24acc94a68c4f6dda2724 [sys.catalog]: This master's current role is: LEADER
I20260808 22:22:28.904589 23263 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:22:28.907953 23263 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:22:28.908640 23263 catalog_manager.cc:1259] Loaded cluster ID: 2fda19fa9dcb4b3f87d21c69c0529ba2
I20260808 22:22:28.908710 23263 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:22:28.911005 23263 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:22:28.911677 23263 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5c245739a3c24acc94a68c4f6dda2724: Loaded TSK: 0
I20260808 22:22:28.912942 23263 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 22:22:28.980692 23253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:22:28.980971 23253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:22:28.981041 23253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:22:28.985036 23253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:22:28.985088 23253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:22:28.985175 23253 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.193
I20260808 22:22:28.990216 23253 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:40251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/krb5kdc/oryx_127.9.176.193.keytab
--principal=oryx/127.9.176.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.193:33877
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.176.193
--webserver_port=37497
--tserver_master_addrs=127.9.176.254:44973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23253
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.193
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:22:28.991114 23253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:22:28.992064 23253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:22:29.001195 23288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:29.001336 23291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:29.001223 23289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:22:29.001569 23253 server_base.cc:1034] running on GCE node
I20260808 22:22:29.001958 23253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:22:29.002586 23253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:22:29.003764 23253 hybrid_clock.cc:648] HybridClock initialized: now 1786227749003745 us; error 35 us; skew 500 ppm
Aug 08 22:22:29 dist-test-slave-ncp9 krb5kdc[22322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227749, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.176.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:22:29.006650 23253 init.cc:377] Logged in from keytab as oryx/127.9.176.193@KRBTEST.COM (short username oryx)
I20260808 22:22:29.007495 23253 webserver.cc:460] Webserver started at http://127.9.176.193:37497/ using document root <none> and password file <none>
I20260808 22:22:29.008001 23253 fs_manager.cc:359] Metadata directory not provided
I20260808 22:22:29.008075 23253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:22:29.011770 23253 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260808 22:22:29.013753 23302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:29.014894 23253 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 22:22:29.014987 23253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/ts-0/wal
uuid: "a09819ea0dac4387a5674f5ed473aa19"
format_stamp: "Formatted at 2026-08-08 22:22:28 on dist-test-slave-ncp9"
server_key: "3fe50f25e17755b4c79b07ccce5a6733"
server_key_iv: "5d32c017ed7876341c43415ad94a9e84"
server_key_version: "encryptionkey@0"
I20260808 22:22:29.015509 23253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:22:29.033958 23253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:22:29.036798 23253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:22:29.036979 23253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:22:29.037552 23253 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:22:29.038368 23253 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:22:29.038417 23253 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:29.038460 23253 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:22:29.038488 23253 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:29.047482 23253 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.193:33877
I20260808 22:22:29.047503 23476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.193:33877 every 8 connection(s)
I20260808 22:22:29.048460 23253 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
I20260808 22:22:29.054388 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 23253
I20260808 22:22:29.054800 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.194:36185
--local_ip_for_outbound_sockets=127.9.176.194
--tserver_master_addrs=127.9.176.254:44973
--webserver_port=46445
--webserver_interface=127.9.176.194
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.194.keytab
--principal=kudu/127.9.176.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:40251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.9.176.194
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/krb5kdc/oryx_127.9.176.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 22:22:29 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.193@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:22:29 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.193@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
W20260808 22:22:29.061269 23480 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:22:29.050457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.193:53391 (local address 127.9.176.254:44973)
0808 22:22:29.050932 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:29.050952 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:29.051351 (+ 399us) server_negotiation.cc:407] Connection header received
0808 22:22:29.052339 (+ 988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:29.052350 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:29.052632 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:29.052930 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:29.053967 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:29.055071 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:29.056268 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:29.056756 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:29.056882 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:29.060570 (+ 3688us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.176.193:53391: BlockingRecv error: recv got EOF from 127.9.176.193:53391 (error 108)
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":113,"threads_started":1}
I20260808 22:22:29.061363 23481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:29.050553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:44973 (local address 127.9.176.193:53391)
0808 22:22:29.051200 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:29.051240 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:29.051978 (+ 738us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:29.053013 (+ 1035us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:29.053019 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:29.053375 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:29.053829 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:29.053842 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:29.055211 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:29.055214 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:29.056100 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:29.056115 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:29.056948 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:29.056953 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:29.057147 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:29.057734 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:29.057753 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:29.060358 (+ 2605us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.176.254:44973: Server oryx/127.9.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":124,"threads_started":1}
W20260808 22:22:29.061873 23478 heartbeater.cc:644] Failed to heartbeat to 127.9.176.254:44973 (0 consecutive failures): Not authorized: Failed to ping master at 127.9.176.254:44973: Client connection negotiation failed: client connection to 127.9.176.254:44973: Server oryx/127.9.176.254@KRBTEST.COM not found in Kerberos database
Aug 08 22:22:29 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.193@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:22:29 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.193@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:22:29.067085 23481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:29.062309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:44973 (local address 127.9.176.193:52813)
0808 22:22:29.062402 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:29.062415 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:29.062487 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:29.062727 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:29.062730 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:29.062872 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:29.062960 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:29.062965 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:29.063660 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:29.063663 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:29.064211 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:29.064220 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:29.064631 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:29.064635 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:29.064726 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:29.065234 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:29.065245 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:29.066974 (+ 1729us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.176.254:44973: Server oryx/127.9.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":28}
W20260808 22:22:29.067389 23480 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:22:29.062413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.193:52813 (local address 127.9.176.254:44973)
0808 22:22:29.062541 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:29.062554 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:29.062566 (+ 12us) server_negotiation.cc:407] Connection header received
0808 22:22:29.062599 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:29.062601 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:29.062636 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:29.062699 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:29.063064 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:29.063561 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:29.064320 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:29.064529 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:29.064618 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:29.067120 (+ 2502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.176.193:52813: BlockingRecv error: recv got EOF from 127.9.176.193:52813 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Aug 08 22:22:29 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.193@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:22:29 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.193@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:22:29.073246 23481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:29.067525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:44973 (local address 127.9.176.193:58573)
0808 22:22:29.067626 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:29.067639 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:29.067707 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:29.068016 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:29.068019 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:29.068142 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:29.068210 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:29.068215 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:29.068875 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:29.068878 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:29.069406 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:29.069415 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:29.069862 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:29.069868 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:29.069968 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:29.070996 (+ 1028us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:29.071016 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:29.073141 (+ 2125us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.176.254:44973: Server oryx/127.9.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":37}
W20260808 22:22:29.073407 23478 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 22:22:29.073522 23480 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:22:29.067756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.193:58573 (local address 127.9.176.254:44973)
0808 22:22:29.067847 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:29.067850 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:29.067860 (+ 10us) server_negotiation.cc:407] Connection header received
0808 22:22:29.067887 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:29.067889 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:29.067920 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:29.067977 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:29.068313 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:29.068778 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:29.069511 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:29.069740 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:29.069831 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:29.073268 (+ 3437us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.176.193:58573: BlockingRecv error: recv got EOF from 127.9.176.193:58573 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
W20260808 22:22:29.166648 23484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:22:29.166906 23484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:22:29.166963 23484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:22:29.170853 23484 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:22:29.170907 23484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:22:29.170982 23484 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.194
I20260808 22:22:29.176023 23484 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:40251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/krb5kdc/oryx_127.9.176.194.keytab
--principal=oryx/127.9.176.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.194:36185
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.176.194
--webserver_port=46445
--tserver_master_addrs=127.9.176.254:44973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23484
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.194
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:22:29.177037 23484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:22:29.178058 23484 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 22:22:29.187517 23484 server_base.cc:1034] running on GCE node
W20260808 22:22:29.187521 23494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:29.187532 23493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:29.187670 23496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:22:29.188153 23484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:22:29.188755 23484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:22:29.189911 23484 hybrid_clock.cc:648] HybridClock initialized: now 1786227749189906 us; error 48 us; skew 500 ppm
Aug 08 22:22:29 dist-test-slave-ncp9 krb5kdc[22322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227749, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.176.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:22:29.192500 23484 init.cc:377] Logged in from keytab as oryx/127.9.176.194@KRBTEST.COM (short username oryx)
I20260808 22:22:29.193372 23484 webserver.cc:460] Webserver started at http://127.9.176.194:46445/ using document root <none> and password file <none>
I20260808 22:22:29.193900 23484 fs_manager.cc:359] Metadata directory not provided
I20260808 22:22:29.193981 23484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:22:29.197726 23484 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260808 22:22:29.199625 23504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:29.200750 23484 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 22:22:29.200855 23484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/ts-1/wal
uuid: "3c4df74acdef4954b44781bc97baa3c2"
format_stamp: "Formatted at 2026-08-08 22:22:28 on dist-test-slave-ncp9"
server_key: "7216e2925575d79aa4af758342b4bb28"
server_key_iv: "f97392a55ab2d8ee4183a476ad5c5de6"
server_key_version: "encryptionkey@0"
I20260808 22:22:29.201398 23484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:22:29.214712 23484 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:22:29.217706 23484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:22:29.217880 23484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:22:29.218473 23484 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:22:29.219349 23484 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:22:29.219403 23484 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:29.219457 23484 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:22:29.219484 23484 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:29.227696 23484 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.194:36185
I20260808 22:22:29.227725 23675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.194:36185 every 8 connection(s)
I20260808 22:22:29.228655 23484 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
I20260808 22:22:29.232036 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 23484
I20260808 22:22:29.232462 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.195:42469
--local_ip_for_outbound_sockets=127.9.176.195
--tserver_master_addrs=127.9.176.254:44973
--webserver_port=44475
--webserver_interface=127.9.176.195
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.195.keytab
--principal=kudu/127.9.176.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:40251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.9.176.195
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/krb5kdc/oryx_127.9.176.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 22:22:29 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.194@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:22:29 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.194@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
W20260808 22:22:29.239981 23480 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:22:29.230372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.194:37473 (local address 127.9.176.254:44973)
0808 22:22:29.230505 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:29.230509 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:29.231527 (+ 1018us) server_negotiation.cc:407] Connection header received
0808 22:22:29.232403 (+ 876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:29.232407 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:29.232468 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:29.232562 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:29.233805 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:29.234425 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:29.235964 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:29.236172 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:29.236242 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:29.239880 (+ 3638us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.176.194:37473: BlockingRecv error: recv got EOF from 127.9.176.194:37473 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 22:22:29.240787 23681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:29.230747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:44973 (local address 127.9.176.194:37473)
0808 22:22:29.231379 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:29.231418 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:29.232185 (+ 767us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:29.232764 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:29.232773 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:29.233179 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:29.233653 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:29.233667 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:29.234556 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:29.234561 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:29.235850 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:29.235862 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:29.236318 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:29.236322 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:29.236505 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:29.237153 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:29.237174 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:29.239772 (+ 2598us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.176.254:44973: Server oryx/127.9.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":115,"threads_started":1}
W20260808 22:22:29.241281 23678 heartbeater.cc:644] Failed to heartbeat to 127.9.176.254:44973 (0 consecutive failures): Not authorized: Failed to ping master at 127.9.176.254:44973: Client connection negotiation failed: client connection to 127.9.176.254:44973: Server oryx/127.9.176.254@KRBTEST.COM not found in Kerberos database
Aug 08 22:22:29 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.194@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:22:29 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.194@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:22:29.246793 23681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:29.241681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:44973 (local address 127.9.176.194:52049)
0808 22:22:29.241807 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:29.241822 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:29.241919 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:29.242146 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:29.242150 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:29.242321 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:29.242442 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:29.242450 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:29.243117 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:29.243120 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:29.243989 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:29.244001 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:29.244421 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:29.244425 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:29.244520 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:29.244967 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:29.244981 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:29.246674 (+ 1693us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.176.254:44973: Server oryx/127.9.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":48}
W20260808 22:22:29.246876 23480 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:22:29.241793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.194:52049 (local address 127.9.176.254:44973)
0808 22:22:29.241900 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:29.241903 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:29.241916 (+ 13us) server_negotiation.cc:407] Connection header received
0808 22:22:29.242009 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:29.242011 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:29.242044 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:29.242104 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:29.242539 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:29.243015 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:29.244093 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:29.244317 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:29.244389 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:29.246788 (+ 2399us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.176.194:52049: BlockingRecv error: recv got EOF from 127.9.176.194:52049 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Aug 08 22:22:29 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.194@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:22:29 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.194@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:22:29.252120 23681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:29.247410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:44973 (local address 127.9.176.194:41327)
0808 22:22:29.247556 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:29.247570 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:29.247628 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:29.247823 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:29.247825 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:29.247940 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:29.248005 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:29.248009 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:29.248988 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:29.248990 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:29.249486 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:29.249493 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:29.250005 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:29.250007 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:29.250058 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:29.250448 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:29.250460 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:29.252005 (+ 1545us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.176.254:44973: Server oryx/127.9.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":91}
W20260808 22:22:29.252295 23480 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:22:29.247505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.194:41327 (local address 127.9.176.254:44973)
0808 22:22:29.247619 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:29.247623 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:29.247636 (+ 13us) server_negotiation.cc:407] Connection header received
0808 22:22:29.247697 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:29.247700 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:29.247738 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:29.247815 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:29.248089 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:29.248861 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:29.249588 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:29.249920 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:29.250011 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:29.252109 (+ 2098us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.176.194:41327: BlockingRecv error: recv got EOF from 127.9.176.194:41327 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260808 22:22:29.252432 23678 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 22:22:29.344762 23683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:22:29.345036 23683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:22:29.345104 23683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:22:29.349057 23683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:22:29.349107 23683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:22:29.349175 23683 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.195
I20260808 22:22:29.354192 23683 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:40251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/krb5kdc/oryx_127.9.176.195.keytab
--principal=oryx/127.9.176.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.195:42469
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.176.195
--webserver_port=44475
--tserver_master_addrs=127.9.176.254:44973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23683
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.195
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:22:29.355118 23683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:22:29.356065 23683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:22:29.365056 23698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:29.365062 23697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:29.365070 23700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:22:29.365329 23683 server_base.cc:1034] running on GCE node
I20260808 22:22:29.365713 23683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:22:29.366289 23683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:22:29.367439 23683 hybrid_clock.cc:648] HybridClock initialized: now 1786227749367423 us; error 26 us; skew 500 ppm
Aug 08 22:22:29 dist-test-slave-ncp9 krb5kdc[22322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227749, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.176.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:22:29.369994 23683 init.cc:377] Logged in from keytab as oryx/127.9.176.195@KRBTEST.COM (short username oryx)
I20260808 22:22:29.370800 23683 webserver.cc:460] Webserver started at http://127.9.176.195:44475/ using document root <none> and password file <none>
I20260808 22:22:29.371289 23683 fs_manager.cc:359] Metadata directory not provided
I20260808 22:22:29.371366 23683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:22:29.375422 23683 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260808 22:22:29.378242 23712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:29.379354 23683 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 22:22:29.379458 23683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/ts-2/wal
uuid: "844363fb7a8f41efb89a2f276c41ba15"
format_stamp: "Formatted at 2026-08-08 22:22:28 on dist-test-slave-ncp9"
server_key: "1c1692659de51d73e5787772867df48f"
server_key_iv: "514b00de909e582070f4c791b4f577ce"
server_key_version: "encryptionkey@0"
I20260808 22:22:29.379963 23683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:22:29.399307 23683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:22:29.401986 23683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:22:29.402149 23683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:22:29.402740 23683 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:22:29.403515 23683 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:22:29.403569 23683 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:29.403622 23683 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:22:29.403646 23683 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:29.412221 23683 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.195:42469
I20260808 22:22:29.412256 23903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.195:42469 every 8 connection(s)
I20260808 22:22:29.413211 23683 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
I20260808 22:22:29.419727 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 23683
Aug 08 22:22:29 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.195@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:22:29 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.195@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
W20260808 22:22:29.423643 23480 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:22:29.414914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.195:37153 (local address 127.9.176.254:44973)
0808 22:22:29.415031 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:29.415034 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:29.416033 (+ 999us) server_negotiation.cc:407] Connection header received
0808 22:22:29.416878 (+ 845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:29.416881 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:29.416963 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:29.417037 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:29.418152 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:29.418751 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:29.420066 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:29.420291 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:29.420392 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:29.423512 (+ 3120us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.176.195:37153: BlockingRecv error: recv got EOF from 127.9.176.195:37153 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20260808 22:22:29.424397 23906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:29.415282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:44973 (local address 127.9.176.195:37153)
0808 22:22:29.415897 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:29.415935 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:29.416684 (+ 749us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:29.417216 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:29.417223 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:29.417547 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:29.417990 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:29.418003 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:29.418866 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:29.418868 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:29.419946 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:29.419958 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:29.420439 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:29.420444 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:29.420615 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:29.421225 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:29.421241 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:29.423388 (+ 2147us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.176.254:44973: Server oryx/127.9.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":110,"threads_started":1}
W20260808 22:22:29.424932 23904 heartbeater.cc:644] Failed to heartbeat to 127.9.176.254:44973 (0 consecutive failures): Not authorized: Failed to ping master at 127.9.176.254:44973: Client connection negotiation failed: client connection to 127.9.176.254:44973: Server oryx/127.9.176.254@KRBTEST.COM not found in Kerberos database
I20260808 22:22:29.428782 23910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:29.420688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33722 (local address 127.9.176.254:44973)
0808 22:22:29.420954 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:29.420958 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:29.420970 (+ 12us) server_negotiation.cc:407] Connection header received
0808 22:22:29.421000 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:29.421002 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:29.421035 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:29.421107 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:29.421904 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:29.422459 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:29.423332 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:29.423552 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:29.423636 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:29.425017 (+ 1381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:29.425024 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:29.425041 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:22:29.425082 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:29.426863 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:29.427258 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:29.427264 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:29.427270 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:29.427418 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:29.427633 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:29.427636 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:29.427637 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:29.428004 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:29.428187 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:29.428384 (+ 197us) server_negotiation.cc:299] Negotiation successful
0808 22:22:29.428629 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":93,"threads_started":1}
Aug 08 22:22:29 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.195@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:22:29 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.195@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:22:29.430233 23906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:29.425344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:44973 (local address 127.9.176.195:44357)
0808 22:22:29.425677 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:29.425687 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:29.425753 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:29.425956 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:29.425959 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:29.426120 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:29.426228 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:29.426236 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:29.426908 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:29.426911 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:29.427611 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:29.427623 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:29.428013 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:29.428017 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:29.428120 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:29.428531 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:29.428545 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:29.430155 (+ 1610us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.176.254:44973: Server oryx/127.9.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":271}
W20260808 22:22:29.430371 23480 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:22:29.425398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.195:44357 (local address 127.9.176.254:44973)
0808 22:22:29.425494 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:29.425497 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:29.425736 (+ 239us) server_negotiation.cc:407] Connection header received
0808 22:22:29.425814 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:29.425817 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:29.425860 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:29.425936 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:29.426343 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:29.426807 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:29.427714 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:29.427914 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:29.427982 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:29.430292 (+ 2310us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.176.195:44357: BlockingRecv error: recv got EOF from 127.9.176.195:44357 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Aug 08 22:22:29 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.195@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:22:29 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.195@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:22:29.435127 23906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:29.430983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:44973 (local address 127.9.176.195:44599)
0808 22:22:29.431133 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:29.431145 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:29.431214 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:29.431411 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:29.431413 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:29.431547 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:29.431608 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:29.431613 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:29.432264 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:29.432267 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:29.432761 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:29.432768 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:29.433214 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:29.433218 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:29.433290 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:29.433594 (+ 304us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:29.433603 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:29.435016 (+ 1413us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.176.254:44973: Server oryx/127.9.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":93}
W20260808 22:22:29.435216 23480 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:22:29.431063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.195:44599 (local address 127.9.176.254:44973)
0808 22:22:29.431207 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:29.431210 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:29.431224 (+ 14us) server_negotiation.cc:407] Connection header received
0808 22:22:29.431289 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:29.431292 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:29.431328 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:29.431388 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:29.431708 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:29.432169 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:29.432864 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:29.433119 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:29.433203 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:29.435134 (+ 1931us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.176.195:44599: BlockingRecv error: recv got EOF from 127.9.176.195:44599 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260808 22:22:29.435354 23904 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 08 22:22:30 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.193@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:22:30 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.193@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:22:30.079497 23914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:30.073999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:44973 (local address 127.9.176.193:35465)
0808 22:22:30.074253 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:30.074268 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:30.074366 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:30.074833 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:30.074836 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:30.075020 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:30.075132 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:30.075138 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:30.076024 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:30.076027 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:30.076607 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:30.076616 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:30.077118 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:30.077122 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:30.077234 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:30.077690 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:30.077702 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:30.079358 (+ 1656us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.176.254:44973: Server oryx/127.9.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":78,"threads_started":1}
W20260808 22:22:30.079598 23915 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:22:30.074129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.193:35465 (local address 127.9.176.254:44973)
0808 22:22:30.074474 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:30.074478 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:30.074493 (+ 15us) server_negotiation.cc:407] Connection header received
0808 22:22:30.074537 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:30.074539 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:30.074590 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:30.074681 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:30.075256 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:30.075916 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:30.076719 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:30.077006 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:30.077091 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:30.079482 (+ 2391us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.176.193:35465: BlockingRecv error: recv got EOF from 127.9.176.193:35465 (error 108)
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":127,"threads_started":1}
Aug 08 22:22:30 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.194@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:22:30 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.194@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:22:30.257889 23918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:30.252998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:44973 (local address 127.9.176.194:39527)
0808 22:22:30.253248 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:30.253262 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:30.253359 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:30.253569 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:30.253571 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:30.253743 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:30.253840 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:30.253845 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:30.254518 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:30.254520 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:30.255093 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:30.255100 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:30.255455 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:30.255457 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:30.255513 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:30.256013 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:30.256025 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:30.257771 (+ 1746us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.176.254:44973: Server oryx/127.9.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":72,"threads_started":1}
W20260808 22:22:30.258009 23915 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:22:30.253050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.194:39527 (local address 127.9.176.254:44973)
0808 22:22:30.253203 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:30.253208 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:30.253321 (+ 113us) server_negotiation.cc:407] Connection header received
0808 22:22:30.253440 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:30.253442 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:30.253486 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:30.253541 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:30.253920 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:30.254439 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:30.255177 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:30.255383 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:30.255445 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:30.257905 (+ 2460us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.176.194:39527: BlockingRecv error: recv got EOF from 127.9.176.194:39527 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Aug 08 22:22:30 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.195@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:22:30 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.195@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:22:30.441067 23919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:30.435864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:44973 (local address 127.9.176.195:45361)
0808 22:22:30.436114 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:30.436129 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:30.436237 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:30.436449 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:30.436451 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:30.436619 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:30.436712 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:30.436717 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:30.437405 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:30.437407 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:30.437973 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:30.437980 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:30.438448 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:30.438450 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:30.438507 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:30.438981 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:30.438993 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:30.440934 (+ 1941us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.176.254:44973: Server oryx/127.9.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":84,"threads_started":1}
W20260808 22:22:30.441138 23915 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:22:30.435931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.195:45361 (local address 127.9.176.254:44973)
0808 22:22:30.436066 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:30.436070 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:30.436203 (+ 133us) server_negotiation.cc:407] Connection header received
0808 22:22:30.436309 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:30.436315 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:30.436360 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:30.436425 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:30.436789 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:30.437331 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:30.438097 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:30.438345 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:30.438443 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:30.441036 (+ 2593us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.176.195:45361: BlockingRecv error: recv got EOF from 127.9.176.195:45361 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Aug 08 22:22:31 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.193@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:22:31 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.193@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:22:31.085294 23920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:31.080198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:44973 (local address 127.9.176.193:60841)
0808 22:22:31.080453 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:31.080467 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:31.080556 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:31.080765 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:31.080768 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:31.080997 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:31.081106 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:31.081113 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:31.081851 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:31.081853 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:31.082420 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:31.082426 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:31.082867 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:31.082870 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:31.082935 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:31.083426 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:31.083440 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:31.085180 (+ 1740us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.176.254:44973: Server oryx/127.9.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":88,"threads_started":1}
W20260808 22:22:31.085422 23921 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:22:31.080245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.193:60841 (local address 127.9.176.254:44973)
0808 22:22:31.080516 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:31.080519 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:31.080533 (+ 14us) server_negotiation.cc:407] Connection header received
0808 22:22:31.080636 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:31.080638 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:31.080679 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:31.080742 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:31.081210 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:31.081764 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:31.082498 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:31.082769 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:31.082844 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:31.085307 (+ 2463us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.176.193:60841: BlockingRecv error: recv got EOF from 127.9.176.193:60841 (error 108)
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":59,"threads_started":1}
Aug 08 22:22:31 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.194@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:22:31 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.194@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:22:31.263463 23922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:31.258611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:44973 (local address 127.9.176.194:44241)
0808 22:22:31.258877 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:31.258891 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:31.258998 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:31.259265 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:31.259267 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:31.259442 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:31.259540 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:31.259546 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:31.260227 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:31.260229 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:31.260774 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:31.260780 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:31.261165 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:31.261168 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:31.261227 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:31.261671 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:31.261682 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:31.263330 (+ 1648us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.176.254:44973: Server oryx/127.9.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":71,"threads_started":1}
W20260808 22:22:31.263573 23921 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:22:31.258678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.194:44241 (local address 127.9.176.254:44973)
0808 22:22:31.258812 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:31.258816 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:31.258958 (+ 142us) server_negotiation.cc:407] Connection header received
0808 22:22:31.259083 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:31.259086 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:31.259143 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:31.259216 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:31.259627 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:31.260151 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:31.260851 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:31.261080 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:31.261149 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:31.263458 (+ 2309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.176.194:44241: BlockingRecv error: recv got EOF from 127.9.176.194:44241 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Aug 08 22:22:31 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.195@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:22:31 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.195@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:22:31.448462 23923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:31.441750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:44973 (local address 127.9.176.195:32811)
0808 22:22:31.441999 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:31.442012 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:31.442106 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:31.442323 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:31.442325 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:31.442492 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:31.442612 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:31.442619 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:31.443658 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:31.443662 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:31.444522 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:31.444533 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:31.445197 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:31.445201 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:31.445292 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:31.445861 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:31.445877 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:31.448289 (+ 2412us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.176.254:44973: Server oryx/127.9.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":68,"threads_started":1}
W20260808 22:22:31.448573 23921 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:22:31.441878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.195:32811 (local address 127.9.176.254:44973)
0808 22:22:31.442015 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:31.442018 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:31.442068 (+ 50us) server_negotiation.cc:407] Connection header received
0808 22:22:31.442191 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:31.442193 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:31.442235 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:31.442298 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:31.442731 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:31.443559 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:31.444693 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:31.445072 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:31.445202 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:31.448458 (+ 3256us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.176.195:32811: BlockingRecv error: recv got EOF from 127.9.176.195:32811 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 08 22:22:32 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.193@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:22:32 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.193@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:22:32.091037 23926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:32.085961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:44973 (local address 127.9.176.193:53935)
0808 22:22:32.086232 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:32.086245 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:32.086330 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:32.086546 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:32.086548 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:32.086720 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:32.086816 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:32.086821 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:32.087528 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:32.087531 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:32.088119 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:32.088127 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:32.088568 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:32.088572 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:32.088648 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:32.089132 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:32.089144 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:32.090911 (+ 1767us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.176.254:44973: Server oryx/127.9.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":105,"threads_started":1}
W20260808 22:22:32.091151 23927 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:22:32.086025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.193:53935 (local address 127.9.176.254:44973)
0808 22:22:32.086266 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:32.086269 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:32.086284 (+ 15us) server_negotiation.cc:407] Connection header received
0808 22:22:32.086404 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:32.086406 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:32.086446 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:32.086519 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:32.086898 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:32.087428 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:32.088201 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:32.088459 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:32.088533 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:32.091040 (+ 2507us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.176.193:53935: BlockingRecv error: recv got EOF from 127.9.176.193:53935 (error 108)
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":87,"threads_started":1}
Aug 08 22:22:32 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.194@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:22:32 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.194@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:22:32.268927 23931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:32.264123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:44973 (local address 127.9.176.194:36727)
0808 22:22:32.264394 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:32.264407 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:32.264498 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:32.264683 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:32.264685 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:32.264854 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:32.264975 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:32.264981 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:32.265653 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:32.265655 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:32.266203 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:32.266209 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:32.266563 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:32.266566 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:32.266623 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:32.267039 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:32.267052 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:32.268785 (+ 1733us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.176.254:44973: Server oryx/127.9.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":80,"threads_started":1}
W20260808 22:22:32.269052 23927 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:22:32.264207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.194:36727 (local address 127.9.176.254:44973)
0808 22:22:32.264328 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:32.264331 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:32.264464 (+ 133us) server_negotiation.cc:407] Connection header received
0808 22:22:32.264561 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:32.264563 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:32.264602 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:32.264671 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:32.265059 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:32.265574 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:32.266282 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:32.266483 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:32.266550 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:32.268950 (+ 2400us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.176.194:36727: BlockingRecv error: recv got EOF from 127.9.176.194:36727 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Aug 08 22:22:32 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.195@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:22:32 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.195@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:22:32.454898 23932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:32.449495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:44973 (local address 127.9.176.195:58177)
0808 22:22:32.449741 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:32.449753 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:32.449844 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:32.450108 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:32.450111 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:32.450364 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:32.450494 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:32.450502 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:32.451372 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:32.451375 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:32.451941 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:32.451950 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:32.452342 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:32.452345 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:32.452403 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:32.452846 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:32.452856 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:32.454789 (+ 1933us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.176.254:44973: Server oryx/127.9.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":82,"threads_started":1}
W20260808 22:22:32.455029 23927 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:22:32.449558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.195:58177 (local address 127.9.176.254:44973)
0808 22:22:32.449683 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:32.449687 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:32.449811 (+ 124us) server_negotiation.cc:407] Connection header received
0808 22:22:32.449928 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:32.449930 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:32.449986 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:32.450061 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:32.450636 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:32.451269 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:32.452039 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:32.452264 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:32.452333 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:32.454912 (+ 2579us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.176.195:58177: BlockingRecv error: recv got EOF from 127.9.176.195:58177 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Aug 08 22:22:33 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.193@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:22:33 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.193@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:22:33.099598 23935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:33.091780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:44973 (local address 127.9.176.193:35583)
0808 22:22:33.092104 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:33.092120 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:33.092231 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:33.092486 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:33.092489 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:33.092699 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:33.092809 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:33.092816 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:33.093615 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:33.093618 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:33.094179 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:33.094187 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:33.094621 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:33.094624 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:33.094711 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:33.097722 (+ 3011us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:33.097733 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:33.099483 (+ 1750us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.176.254:44973: Server oryx/127.9.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":81,"threads_started":1}
W20260808 22:22:33.099776 23937 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:22:33.091830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.193:35583 (local address 127.9.176.254:44973)
0808 22:22:33.092093 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:33.092098 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:33.092192 (+ 94us) server_negotiation.cc:407] Connection header received
0808 22:22:33.092319 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:33.092322 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:33.092371 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:33.092444 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:33.092951 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:33.093510 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:33.094288 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:33.094519 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:33.094605 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:33.099638 (+ 5033us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.176.193:35583: BlockingRecv error: recv got EOF from 127.9.176.193:35583 (error 108)
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":77,"threads_started":1}
Aug 08 22:22:33 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.194@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:22:33 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.194@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:22:33.274777 23938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:33.269648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:44973 (local address 127.9.176.194:46933)
0808 22:22:33.269933 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:33.269947 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:33.270065 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:33.270308 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:33.270310 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:33.270507 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:33.270607 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:33.270613 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:33.271359 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:33.271363 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:33.271921 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:33.271930 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:33.272369 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:33.272373 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:33.272453 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:33.272974 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:33.272985 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:33.274661 (+ 1676us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.176.254:44973: Server oryx/127.9.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":105,"threads_started":1}
W20260808 22:22:33.274904 23937 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:22:33.269740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.194:46933 (local address 127.9.176.254:44973)
0808 22:22:33.269897 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:33.269902 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:33.270014 (+ 112us) server_negotiation.cc:407] Connection header received
0808 22:22:33.270152 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:33.270155 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:33.270204 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:33.270267 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:33.270726 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:33.271253 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:33.272038 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:33.272267 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:33.272355 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:33.274798 (+ 2443us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.176.194:46933: BlockingRecv error: recv got EOF from 127.9.176.194:46933 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 08 22:22:33 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.195@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:22:33 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.195@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:22:33.460732 23940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:33.455660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:44973 (local address 127.9.176.195:55847)
0808 22:22:33.455921 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:33.455934 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:33.456036 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:33.456258 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:33.456260 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:33.456431 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:33.456528 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:33.456533 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:33.457335 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:33.457338 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:33.457905 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:33.457914 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:33.458290 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:33.458293 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:33.458353 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:33.458822 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:33.458833 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:33.460571 (+ 1738us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.176.254:44973: Server oryx/127.9.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":77,"threads_started":1}
W20260808 22:22:33.460932 23937 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:22:33.455726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.195:55847 (local address 127.9.176.254:44973)
0808 22:22:33.455859 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:33.455862 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:33.455995 (+ 133us) server_negotiation.cc:407] Connection header received
0808 22:22:33.456103 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:33.456105 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:33.456150 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:33.456231 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:33.456619 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:33.457209 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:33.457991 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:33.458203 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:33.458269 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:33.460793 (+ 2524us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.176.195:55847: BlockingRecv error: recv got EOF from 127.9.176.195:55847 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 08 22:22:34 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.193@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:22:34 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.193@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:22:34.105551 23941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:34.100364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:44973 (local address 127.9.176.193:56673)
0808 22:22:34.100628 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:34.100641 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:34.100726 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:34.100952 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:34.100954 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:34.101145 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:34.101242 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:34.101247 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:34.101934 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:34.101936 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:34.102530 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:34.102538 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:34.102900 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:34.102902 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:34.102960 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:34.103434 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:34.103444 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:34.105406 (+ 1962us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.176.254:44973: Server oryx/127.9.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":73,"threads_started":1}
W20260808 22:22:34.105649 23942 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:22:34.100439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.193:56673 (local address 127.9.176.254:44973)
0808 22:22:34.100686 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:34.100690 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:34.100704 (+ 14us) server_negotiation.cc:407] Connection header received
0808 22:22:34.100792 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:34.100794 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:34.100835 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:34.100936 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:34.101326 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:34.101851 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:34.102609 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:34.102824 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:34.102891 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:34.105536 (+ 2645us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.176.193:56673: BlockingRecv error: recv got EOF from 127.9.176.193:56673 (error 108)
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":102,"threads_started":1}
Aug 08 22:22:34 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.194@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:22:34 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.194@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:22:34.280747 23943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:34.275585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:44973 (local address 127.9.176.194:53481)
0808 22:22:34.275905 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:34.275919 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:34.276027 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:34.276278 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:34.276281 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:34.276520 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:34.276643 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:34.276651 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:34.277416 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:34.277418 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:34.277982 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:34.277990 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:34.278365 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:34.278367 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:34.278426 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:34.278866 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:34.278877 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:34.280624 (+ 1747us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.176.254:44973: Server oryx/127.9.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":119,"threads_started":1}
W20260808 22:22:34.280916 23942 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:22:34.275721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.194:53481 (local address 127.9.176.254:44973)
0808 22:22:34.275853 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:34.275858 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:34.275998 (+ 140us) server_negotiation.cc:407] Connection header received
0808 22:22:34.276108 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:34.276111 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:34.276167 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:34.276249 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:34.276748 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:34.277324 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:34.278084 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:34.278289 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:34.278353 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:34.280782 (+ 2429us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.176.194:53481: BlockingRecv error: recv got EOF from 127.9.176.194:53481 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Aug 08 22:22:34 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.195@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:22:34 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.195@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:22:34.466585 23944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:34.461568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:44973 (local address 127.9.176.195:54409)
0808 22:22:34.461840 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:34.461852 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:34.461953 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:34.462239 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:34.462241 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:34.462425 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:34.462518 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:34.462523 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:34.463242 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:34.463245 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:34.463800 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:34.463807 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:34.464205 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:34.464207 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:34.464261 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:34.464742 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:34.464753 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:34.466472 (+ 1719us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.176.254:44973: Server oryx/127.9.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":101,"threads_started":1}
W20260808 22:22:34.466805 23942 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:22:34.461637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.195:54409 (local address 127.9.176.254:44973)
0808 22:22:34.461788 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:34.461793 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:34.461914 (+ 121us) server_negotiation.cc:407] Connection header received
0808 22:22:34.462071 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:34.462074 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:34.462128 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:34.462212 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:34.462628 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:34.463163 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:34.463899 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:34.464125 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:34.464198 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:34.466681 (+ 2483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.176.195:54409: BlockingRecv error: recv got EOF from 127.9.176.195:54409 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 08 22:22:35 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.193@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:22:35 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.193@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:22:35.111619 23945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:35.106331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:44973 (local address 127.9.176.193:36807)
0808 22:22:35.106612 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:35.106626 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:35.106720 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:35.106940 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:35.106941 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:35.107124 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:35.107222 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:35.107227 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:35.107917 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:35.107919 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:35.108471 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:35.108478 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:35.109025 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:35.109029 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:35.109118 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:35.109688 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:35.109704 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:35.111495 (+ 1791us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.176.254:44973: Server oryx/127.9.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":91,"threads_started":1}
W20260808 22:22:35.111756 23946 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:22:35.106397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.193:36807 (local address 127.9.176.254:44973)
0808 22:22:35.106620 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:35.106624 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:35.106674 (+ 50us) server_negotiation.cc:407] Connection header received
0808 22:22:35.106791 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:35.106794 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:35.106837 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:35.106915 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:35.107304 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:35.107839 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:35.108598 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:35.108866 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:35.109026 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:35.111628 (+ 2602us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.176.193:36807: BlockingRecv error: recv got EOF from 127.9.176.193:36807 (error 108)
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":72,"threads_started":1}
Aug 08 22:22:35 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.194@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:22:35 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.194@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:22:35.286548 23947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:35.281497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:44973 (local address 127.9.176.194:35153)
0808 22:22:35.281754 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:35.281767 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:35.281875 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:35.282112 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:35.282115 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:35.282338 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:35.282433 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:35.282439 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:35.283203 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:35.283207 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:35.283760 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:35.283768 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:35.284203 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:35.284207 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:35.284291 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:35.284748 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:35.284759 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:35.286400 (+ 1641us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.176.254:44973: Server oryx/127.9.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":90,"threads_started":1}
W20260808 22:22:35.286644 23946 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:22:35.281553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.194:35153 (local address 127.9.176.254:44973)
0808 22:22:35.281696 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:35.281700 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:35.281832 (+ 132us) server_negotiation.cc:407] Connection header received
0808 22:22:35.281959 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:35.281962 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:35.282014 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:35.282078 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:35.282547 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:35.283076 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:35.283876 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:35.284098 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:35.284182 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:35.286545 (+ 2363us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.176.194:35153: BlockingRecv error: recv got EOF from 127.9.176.194:35153 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Aug 08 22:22:35 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.195@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:22:35 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.195@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:22:35.472770 23954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:35.467329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:44973 (local address 127.9.176.195:42199)
0808 22:22:35.467646 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:35.467658 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:35.467768 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:35.468003 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:35.468006 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:35.468210 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:35.468303 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:35.468309 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:35.469160 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:35.469163 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:35.469886 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:35.469899 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:35.470395 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:35.470398 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:35.470461 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:35.470938 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:35.470950 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:35.472659 (+ 1709us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.176.254:44973: Server oryx/127.9.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":90,"threads_started":1}
W20260808 22:22:35.472885 23946 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:22:35.467413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.195:42199 (local address 127.9.176.254:44973)
0808 22:22:35.467544 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:35.467549 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:35.467730 (+ 181us) server_negotiation.cc:407] Connection header received
0808 22:22:35.467852 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:35.467855 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:35.467902 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:35.467984 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:35.468416 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:35.469051 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:35.470019 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:35.470301 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:35.470380 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:35.472783 (+ 2403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.176.195:42199: BlockingRecv error: recv got EOF from 127.9.176.195:42199 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Aug 08 22:22:36 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.193@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:22:36 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.193@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:22:36.117815 23955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:36.112364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:44973 (local address 127.9.176.193:54085)
0808 22:22:36.112671 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:36.112685 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:36.112785 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:36.113059 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:36.113061 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:36.113241 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:36.113337 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:36.113342 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:36.114105 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:36.114108 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:36.114703 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:36.114715 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:36.115214 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:36.115218 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:36.115311 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:36.115887 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:36.115902 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:36.117700 (+ 1798us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.176.254:44973: Server oryx/127.9.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":98,"threads_started":1}
W20260808 22:22:36.117944 23956 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:22:36.112447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.193:54085 (local address 127.9.176.254:44973)
0808 22:22:36.112674 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:36.112678 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:36.112745 (+ 67us) server_negotiation.cc:407] Connection header received
0808 22:22:36.112871 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:36.112873 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:36.112947 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:36.113021 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:36.113421 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:36.113978 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:36.114821 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:36.115085 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:36.115186 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:36.117827 (+ 2641us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.176.193:54085: BlockingRecv error: recv got EOF from 127.9.176.193:54085 (error 108)
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":72,"threads_started":1}
Aug 08 22:22:36 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.194@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:22:36 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.194@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:22:36.293603 23958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:36.287357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:44973 (local address 127.9.176.194:45179)
0808 22:22:36.288583 (+ 1226us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:36.288596 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:36.288699 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:36.289018 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:36.289021 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:36.289230 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:36.289350 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:36.289357 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:36.290116 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:36.290119 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:36.290694 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:36.290703 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:36.291150 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:36.291152 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:36.291211 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:36.291715 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:36.291729 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:36.293488 (+ 1759us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.176.254:44973: Server oryx/127.9.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1142,"thread_start_us":951,"threads_started":1}
W20260808 22:22:36.293749 23956 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:22:36.287427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.194:45179 (local address 127.9.176.254:44973)
0808 22:22:36.287588 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:36.287594 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:36.288682 (+ 1088us) server_negotiation.cc:407] Connection header received
0808 22:22:36.288789 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:36.288792 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:36.288855 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:36.288979 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:36.289455 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:36.290005 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:36.290822 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:36.291058 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:36.291172 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:36.293632 (+ 2460us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.176.194:45179: BlockingRecv error: recv got EOF from 127.9.176.194:45179 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 08 22:22:36 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.195@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:22:36 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.195@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:22:36.478705 23959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:36.473474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:44973 (local address 127.9.176.195:58027)
0808 22:22:36.473779 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:36.473792 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:36.473888 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:36.474094 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:36.474096 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:36.474274 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:36.474369 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:36.474373 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:36.475078 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:36.475080 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:36.475646 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:36.475653 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:36.476008 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:36.476010 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:36.476066 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:36.476526 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:36.476537 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:36.478591 (+ 2054us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.176.254:44973: Server oryx/127.9.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":73,"threads_started":1}
W20260808 22:22:36.478828 23956 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:22:36.473551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.195:58027 (local address 127.9.176.254:44973)
0808 22:22:36.473672 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:36.473676 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:36.473852 (+ 176us) server_negotiation.cc:407] Connection header received
0808 22:22:36.473956 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:36.473958 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:36.474002 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:36.474068 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:36.474456 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:36.474981 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:36.475727 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:36.475932 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:36.475996 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:36.478712 (+ 2716us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.176.195:58027: BlockingRecv error: recv got EOF from 127.9.176.195:58027 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Aug 08 22:22:37 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.193@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:22:37 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.193@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:22:37.124027 23960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:37.118619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:44973 (local address 127.9.176.193:50459)
0808 22:22:37.118874 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:37.118887 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:37.118976 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:37.119229 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:37.119232 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:37.119463 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:37.119565 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:37.119571 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:37.120567 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:37.120569 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:37.121233 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:37.121244 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:37.121650 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:37.121659 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:37.121717 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:37.122191 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:37.122203 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:37.123913 (+ 1710us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.176.254:44973: Server oryx/127.9.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":83,"threads_started":1}
W20260808 22:22:37.124181 23961 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:22:37.118755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.193:50459 (local address 127.9.176.254:44973)
0808 22:22:37.118957 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:37.118960 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:37.118973 (+ 13us) server_negotiation.cc:407] Connection header received
0808 22:22:37.119053 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:37.119055 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:37.119113 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:37.119183 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:37.119681 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:37.120469 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:37.121339 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:37.121564 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:37.121643 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:37.124042 (+ 2399us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.176.193:50459: BlockingRecv error: recv got EOF from 127.9.176.193:50459 (error 108)
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":64,"threads_started":1}
Aug 08 22:22:37 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.194@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:22:37 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.194@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:22:37.299984 23963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:37.294364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:44973 (local address 127.9.176.194:41179)
0808 22:22:37.294621 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:37.294636 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:37.294747 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:37.295014 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:37.295016 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:37.295205 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:37.295322 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:37.295329 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:37.296174 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:37.296177 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:37.296765 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:37.296775 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:37.297209 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:37.297212 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:37.297283 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:37.297801 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:37.297815 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:37.299849 (+ 2034us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.176.254:44973: Server oryx/127.9.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":73,"threads_started":1}
W20260808 22:22:37.300154 23961 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:22:37.294455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.194:41179 (local address 127.9.176.254:44973)
0808 22:22:37.294638 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:37.294642 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:37.294706 (+ 64us) server_negotiation.cc:407] Connection header received
0808 22:22:37.294864 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:37.294868 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:37.294917 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:37.294986 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:37.295413 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:37.296067 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:37.296858 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:37.297109 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:37.297197 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:37.300022 (+ 2825us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.176.194:41179: BlockingRecv error: recv got EOF from 127.9.176.194:41179 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Aug 08 22:22:37 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.195@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:22:37 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.195@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:22:37.484791 23965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:37.479417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:44973 (local address 127.9.176.195:53073)
0808 22:22:37.479704 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:37.479717 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:37.479824 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:37.480123 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:37.480126 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:37.480361 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:37.480477 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:37.480485 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:37.481363 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:37.481365 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:37.481922 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:37.481930 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:37.482322 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:37.482326 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:37.482389 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:37.482875 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:37.482885 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:37.484680 (+ 1795us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.176.254:44973: Server oryx/127.9.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":80,"threads_started":1}
W20260808 22:22:37.484926 23961 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:22:37.479498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.195:53073 (local address 127.9.176.254:44973)
0808 22:22:37.479632 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:37.479637 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:37.479778 (+ 141us) server_negotiation.cc:407] Connection header received
0808 22:22:37.479932 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:37.479936 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:37.479995 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:37.480080 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:37.480598 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:37.481270 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:37.482009 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:37.482233 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:37.482301 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:37.484801 (+ 2500us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.176.195:53073: BlockingRecv error: recv got EOF from 127.9.176.195:53073 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Aug 08 22:22:38 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.193@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:22:38 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.193@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:22:38.133219 23967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:38.124762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:44973 (local address 127.9.176.193:41505)
0808 22:22:38.125050 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:38.125063 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:38.125169 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:38.125392 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:38.125394 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:38.125577 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:38.125679 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:38.125684 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:38.126365 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:38.126367 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:38.126919 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:38.126926 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:38.127398 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:38.127402 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:38.127490 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:38.130869 (+ 3379us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:38.130881 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:38.133071 (+ 2190us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.176.254:44973: Server oryx/127.9.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":76,"threads_started":1}
W20260808 22:22:38.133333 23969 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:22:38.124832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.193:41505 (local address 127.9.176.254:44973)
0808 22:22:38.125095 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:38.125099 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:38.125114 (+ 15us) server_negotiation.cc:407] Connection header received
0808 22:22:38.125246 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:38.125248 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:38.125289 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:38.125364 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:38.125761 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:38.126288 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:38.127032 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:38.127285 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:38.127369 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:38.133218 (+ 5849us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.176.193:41505: BlockingRecv error: recv got EOF from 127.9.176.193:41505 (error 108)
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":99,"threads_started":1}
Aug 08 22:22:38 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.194@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:22:38 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.194@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:22:38.305850 23970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:38.300680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:44973 (local address 127.9.176.194:50087)
0808 22:22:38.300978 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:38.300991 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:38.301090 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:38.301290 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:38.301292 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:38.301466 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:38.301581 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:38.301587 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:38.302275 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:38.302277 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:38.302834 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:38.302842 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:38.303194 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:38.303197 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:38.303255 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:38.303772 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:38.303784 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:38.305703 (+ 1919us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.176.254:44973: Server oryx/127.9.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":95,"threads_started":1}
W20260808 22:22:38.305962 23969 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:22:38.300748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.194:50087 (local address 127.9.176.254:44973)
0808 22:22:38.300873 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:38.300876 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:38.301049 (+ 173us) server_negotiation.cc:407] Connection header received
0808 22:22:38.301155 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:38.301157 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:38.301202 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:38.301265 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:38.301668 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:38.302194 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:38.302913 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:38.303120 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:38.303183 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:38.305863 (+ 2680us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.176.194:50087: BlockingRecv error: recv got EOF from 127.9.176.194:50087 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Aug 08 22:22:38 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.195@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:22:38 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.195@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:22:38.490688 23971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:38.485493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:44973 (local address 127.9.176.195:55223)
0808 22:22:38.485750 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:38.485763 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:38.485869 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:38.486110 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:38.486113 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:38.486315 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:38.486410 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:38.486416 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:38.487166 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:38.487169 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:38.487721 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:38.487729 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:38.488170 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:38.488174 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:38.488258 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:38.488713 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:38.488724 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:38.490508 (+ 1784us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.176.254:44973: Server oryx/127.9.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":82,"threads_started":1}
W20260808 22:22:38.490754 23969 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:22:38.485545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.195:55223 (local address 127.9.176.254:44973)
0808 22:22:38.485670 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:38.485673 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:38.485842 (+ 169us) server_negotiation.cc:407] Connection header received
0808 22:22:38.485955 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:38.485958 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:38.486009 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:38.486073 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:38.486526 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:38.487054 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:38.487835 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:38.488064 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:38.488148 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:38.490646 (+ 2498us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.176.195:55223: BlockingRecv error: recv got EOF from 127.9.176.195:55223 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Aug 08 22:22:39 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.193@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:22:39 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.193@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:22:39.139380 23972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:39.134072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:44973 (local address 127.9.176.193:54095)
0808 22:22:39.134346 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:39.134359 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:39.134453 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:39.134674 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:39.134677 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:39.134855 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:39.134952 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:39.134957 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:39.135694 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:39.135697 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:39.136255 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:39.136262 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:39.136718 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:39.136721 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:39.136807 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:39.137350 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:39.137362 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:39.139264 (+ 1902us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.176.254:44973: Server oryx/127.9.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":92,"threads_started":1}
W20260808 22:22:39.139569 23973 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:22:39.134142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.193:54095 (local address 127.9.176.254:44973)
0808 22:22:39.134368 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:39.134372 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:39.134411 (+ 39us) server_negotiation.cc:407] Connection header received
0808 22:22:39.134528 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:39.134530 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:39.134574 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:39.134648 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:39.135060 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:39.135612 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:39.136351 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:39.136588 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:39.136666 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:39.139428 (+ 2762us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.176.193:54095: BlockingRecv error: recv got EOF from 127.9.176.193:54095 (error 108)
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":70,"threads_started":1}
Aug 08 22:22:39 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.194@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:22:39 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.194@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:22:39.311578 23974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:39.306586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:44973 (local address 127.9.176.194:52969)
0808 22:22:39.306829 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:39.306842 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:39.306934 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:39.307142 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:39.307144 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:39.307327 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:39.307424 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:39.307429 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:39.308142 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:39.308145 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:39.308746 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:39.308756 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:39.309176 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:39.309179 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:39.309251 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:39.309730 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:39.309743 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:39.311439 (+ 1696us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.176.254:44973: Server oryx/127.9.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":71,"threads_started":1}
W20260808 22:22:39.311676 23973 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:22:39.306645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.194:52969 (local address 127.9.176.254:44973)
0808 22:22:39.306771 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:39.306775 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:39.306901 (+ 126us) server_negotiation.cc:407] Connection header received
0808 22:22:39.307004 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:39.307006 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:39.307046 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:39.307112 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:39.307505 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:39.308022 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:39.308846 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:39.309073 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:39.309140 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:39.311575 (+ 2435us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.176.194:52969: BlockingRecv error: recv got EOF from 127.9.176.194:52969 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Aug 08 22:22:39 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.195@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:22:39 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.195@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:22:39.496795 23977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:39.491425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:44973 (local address 127.9.176.195:45621)
0808 22:22:39.491694 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:39.491708 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:39.491793 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:39.491990 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:39.491992 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:39.492180 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:39.492277 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:39.492282 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:39.493035 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:39.493038 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:39.493684 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:39.493697 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:39.494259 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:39.494263 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:39.494345 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:39.494808 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:39.494820 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:39.496641 (+ 1821us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.176.254:44973: Server oryx/127.9.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":82,"threads_started":1}
W20260808 22:22:39.496883 23973 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:22:39.491596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.195:45621 (local address 127.9.176.254:44973)
0808 22:22:39.491738 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:39.491742 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:39.491756 (+ 14us) server_negotiation.cc:407] Connection header received
0808 22:22:39.491861 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:39.491863 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:39.491906 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:39.491966 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:39.492363 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:39.492921 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:39.493822 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:39.494126 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:39.494248 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:39.496782 (+ 2534us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.176.195:45621: BlockingRecv error: recv got EOF from 127.9.176.195:45621 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Aug 08 22:22:40 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.193@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:22:40 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.193@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:22:40.145259 23979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:40.140120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:44973 (local address 127.9.176.193:43823)
0808 22:22:40.140375 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:40.140388 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:40.140475 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:40.140722 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:40.140724 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:40.140927 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:40.141031 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:40.141037 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:40.141743 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:40.141745 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:40.142314 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:40.142322 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:40.142694 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:40.142696 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:40.142751 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:40.143177 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:40.143188 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:40.145094 (+ 1906us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.176.254:44973: Server oryx/127.9.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":74,"threads_started":1}
W20260808 22:22:40.145350 23980 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:22:40.140193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.193:43823 (local address 127.9.176.254:44973)
0808 22:22:40.140417 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:40.140421 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:40.140433 (+ 12us) server_negotiation.cc:407] Connection header received
0808 22:22:40.140569 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:40.140571 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:40.140613 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:40.140697 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:40.141116 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:40.141649 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:40.142397 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:40.142609 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:40.142685 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:40.145235 (+ 2550us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.176.193:43823: BlockingRecv error: recv got EOF from 127.9.176.193:43823 (error 108)
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":75,"threads_started":1}
Aug 08 22:22:40 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.194@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:22:40 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.194@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:22:40.317518 23981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:40.312313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:44973 (local address 127.9.176.194:52547)
0808 22:22:40.312605 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:40.312617 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:40.312727 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:40.312992 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:40.312995 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:40.313195 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:40.313290 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:40.313296 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:40.314056 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:40.314060 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:40.314621 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:40.314630 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:40.315067 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:40.315070 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:40.315150 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:40.315600 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:40.315612 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:40.317401 (+ 1789us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.176.254:44973: Server oryx/127.9.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":87,"threads_started":1}
W20260808 22:22:40.317657 23980 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:22:40.312381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.194:52547 (local address 127.9.176.254:44973)
0808 22:22:40.312534 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:40.312538 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:40.312684 (+ 146us) server_negotiation.cc:407] Connection header received
0808 22:22:40.312812 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:40.312815 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:40.312866 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:40.312951 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:40.313423 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:40.313951 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:40.314735 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:40.314967 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:40.315060 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:40.317535 (+ 2475us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.176.194:52547: BlockingRecv error: recv got EOF from 127.9.176.194:52547 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 08 22:22:40 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.195@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:22:40 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.195@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:22:40.502713 23984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:40.497532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:44973 (local address 127.9.176.195:53037)
0808 22:22:40.497780 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:40.497793 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:40.497892 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:40.498100 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:40.498102 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:40.498293 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:40.498391 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:40.498396 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:40.499082 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:40.499084 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:40.499641 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:40.499647 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:40.500061 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:40.500066 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:40.500156 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:40.500694 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:40.500709 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:40.502590 (+ 1881us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.176.254:44973: Server oryx/127.9.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":77,"threads_started":1}
W20260808 22:22:40.502818 23980 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:22:40.497594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.195:53037 (local address 127.9.176.254:44973)
0808 22:22:40.497725 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:40.497731 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:40.497852 (+ 121us) server_negotiation.cc:407] Connection header received
0808 22:22:40.497958 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:40.497960 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:40.498009 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:40.498077 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:40.498473 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:40.499005 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:40.499718 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:40.499944 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:40.500026 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:40.502715 (+ 2689us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.176.195:53037: BlockingRecv error: recv got EOF from 127.9.176.195:53037 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Aug 08 22:22:41 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.193@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:22:41 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.193@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:22:41.151221 23985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:41.146051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:44973 (local address 127.9.176.193:54197)
0808 22:22:41.146316 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:41.146329 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:41.146425 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:41.146651 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:41.146653 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:41.146830 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:41.146930 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:41.146935 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:41.147626 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:41.147629 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:41.148336 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:41.148347 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:41.148922 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:41.148927 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:41.149015 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:41.149474 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:41.149486 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:41.151107 (+ 1621us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.176.254:44973: Server oryx/127.9.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":88,"threads_started":1}
W20260808 22:22:41.151374 23987 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:22:41.146139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.193:54197 (local address 127.9.176.254:44973)
0808 22:22:41.146377 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:41.146381 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:41.146395 (+ 14us) server_negotiation.cc:407] Connection header received
0808 22:22:41.146503 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:41.146505 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:41.146548 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:41.146623 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:41.147011 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:41.147546 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:41.148451 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:41.148766 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:41.148871 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:41.151242 (+ 2371us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.176.193:54197: BlockingRecv error: recv got EOF from 127.9.176.193:54197 (error 108)
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":61,"threads_started":1}
Aug 08 22:22:41 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.194@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:22:41 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.194@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:22:41.323205 23989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:41.318183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:44973 (local address 127.9.176.194:59651)
0808 22:22:41.318427 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:41.318443 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:41.318542 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:41.318771 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:41.318774 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:41.318952 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:41.319074 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:41.319081 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:41.319770 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:41.319774 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:41.320353 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:41.320361 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:41.320755 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:41.320757 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:41.320817 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:41.321286 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:41.321298 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:41.323090 (+ 1792us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.176.254:44973: Server oryx/127.9.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":78,"threads_started":1}
W20260808 22:22:41.323369 23987 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:22:41.318216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.194:59651 (local address 127.9.176.254:44973)
0808 22:22:41.318364 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:41.318369 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:41.318511 (+ 142us) server_negotiation.cc:407] Connection header received
0808 22:22:41.318628 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:41.318629 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:41.318674 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:41.318737 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:41.319165 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:41.319682 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:41.320450 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:41.320670 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:41.320744 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:41.323243 (+ 2499us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.176.194:59651: BlockingRecv error: recv got EOF from 127.9.176.194:59651 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 08 22:22:41 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.195@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:22:41 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.195@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:22:41.508755 23990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:41.503425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:44973 (local address 127.9.176.195:53627)
0808 22:22:41.503672 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:41.503686 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:41.503783 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:41.503981 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:41.503983 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:41.504187 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:41.504289 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:41.504295 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:41.505181 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:41.505184 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:41.505743 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:41.505751 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:41.506197 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:41.506200 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:41.506265 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:41.506700 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:41.506711 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:41.508604 (+ 1893us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.176.254:44973: Server oryx/127.9.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":92,"threads_started":1}
W20260808 22:22:41.508833 23987 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:22:41.503464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.195:53627 (local address 127.9.176.254:44973)
0808 22:22:41.503594 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:41.503599 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:41.503745 (+ 146us) server_negotiation.cc:407] Connection header received
0808 22:22:41.503846 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:41.503848 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:41.503895 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:41.503957 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:41.504395 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:41.505084 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:41.505832 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:41.506111 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:41.506186 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:41.508734 (+ 2548us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.176.195:53627: BlockingRecv error: recv got EOF from 127.9.176.195:53627 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Aug 08 22:22:42 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.193@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:22:42 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.193@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:22:42.156836 23991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:42.151934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:44973 (local address 127.9.176.193:56669)
0808 22:22:42.152199 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:42.152213 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:42.152303 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:42.152527 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:42.152529 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:42.152719 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:42.152815 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:42.152820 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:42.153518 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:42.153520 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:42.154076 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:42.154084 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:42.154434 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:42.154436 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:42.154491 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:42.154942 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:42.154952 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:42.156721 (+ 1769us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.176.254:44973: Server oryx/127.9.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":97,"threads_started":1}
W20260808 22:22:42.157004 23992 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:22:42.152015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.193:56669 (local address 127.9.176.254:44973)
0808 22:22:42.152239 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:42.152244 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:42.152258 (+ 14us) server_negotiation.cc:407] Connection header received
0808 22:22:42.152382 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:42.152384 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:42.152426 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:42.152506 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:42.152917 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:42.153440 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:42.154157 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:42.154363 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:42.154426 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:42.156853 (+ 2427us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.176.193:56669: BlockingRecv error: recv got EOF from 127.9.176.193:56669 (error 108)
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":62,"threads_started":1}
Aug 08 22:22:42 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.194@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:22:42 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.194@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:22:42.329283 23994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:42.323895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:44973 (local address 127.9.176.194:52681)
0808 22:22:42.324161 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:42.324177 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:42.324303 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:42.324541 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:42.324561 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:42.324762 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:42.324856 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:42.324862 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:42.325636 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:42.325640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:42.326246 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:42.326255 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:42.326684 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:42.326688 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:42.326770 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:42.327261 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:42.327272 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:42.329147 (+ 1875us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.176.254:44973: Server oryx/127.9.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":83,"threads_started":1}
W20260808 22:22:42.329370 23992 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:22:42.324046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.194:52681 (local address 127.9.176.254:44973)
0808 22:22:42.324184 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:42.324195 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:42.324251 (+ 56us) server_negotiation.cc:407] Connection header received
0808 22:22:42.324389 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:42.324392 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:42.324440 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:42.324507 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:42.324993 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:42.325531 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:42.326356 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:42.326582 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:42.326669 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:42.329274 (+ 2605us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.176.194:52681: BlockingRecv error: recv got EOF from 127.9.176.194:52681 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 08 22:22:42 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.195@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:22:42 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.195@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:22:42.514654 23995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:42.509602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:44973 (local address 127.9.176.195:58953)
0808 22:22:42.509864 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:42.509876 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:42.509971 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:42.510168 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:42.510170 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:42.510345 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:42.510443 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:42.510448 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:42.511130 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:42.511132 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:42.511705 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:42.511713 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:42.512113 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:42.512115 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:42.512170 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:42.512621 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:42.512632 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:42.514486 (+ 1854us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.176.254:44973: Server oryx/127.9.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":88,"threads_started":1}
W20260808 22:22:42.514720 23992 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:22:42.509694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.195:58953 (local address 127.9.176.254:44973)
0808 22:22:42.509810 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:42.509813 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:42.509927 (+ 114us) server_negotiation.cc:407] Connection header received
0808 22:22:42.510032 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:42.510034 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:42.510077 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:42.510142 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:42.510531 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:42.511053 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:42.511800 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:42.512034 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:42.512106 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:42.514619 (+ 2513us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.176.195:58953: BlockingRecv error: recv got EOF from 127.9.176.195:58953 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Aug 08 22:22:43 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.193@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:22:43 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.193@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:22:43.165388 23997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:43.157534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:44973 (local address 127.9.176.193:59287)
0808 22:22:43.157806 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:43.157817 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:43.157897 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:43.158150 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:43.158160 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:43.158367 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:43.158497 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:43.158507 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:43.159454 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:43.159458 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:43.160008 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:43.160016 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:43.160377 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:43.160379 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:43.160439 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:43.163459 (+ 3020us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:43.163473 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:43.165276 (+ 1803us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.176.254:44973: Server oryx/127.9.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":69,"threads_started":1}
W20260808 22:22:43.165529 23999 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:22:43.157606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.193:59287 (local address 127.9.176.254:44973)
0808 22:22:43.157894 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:43.157898 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:43.157910 (+ 12us) server_negotiation.cc:407] Connection header received
0808 22:22:43.157958 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:43.157960 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:43.158012 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:43.158120 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:43.158623 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:43.159357 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:43.160093 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:43.160303 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:43.160368 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:43.165408 (+ 5040us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.176.193:59287: BlockingRecv error: recv got EOF from 127.9.176.193:59287 (error 108)
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":68,"threads_started":1}
Aug 08 22:22:43 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.194@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:22:43 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.194@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:22:43.334941 24001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:43.330008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:44973 (local address 127.9.176.194:49123)
0808 22:22:43.330260 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:43.330272 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:43.330370 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:43.330573 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:43.330576 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:43.330755 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:43.330853 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:43.330858 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:43.331518 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:43.331520 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:43.332097 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:43.332105 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:43.332459 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:43.332462 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:43.332519 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:43.333030 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:43.333042 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:43.334778 (+ 1736us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.176.254:44973: Server oryx/127.9.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":79,"threads_started":1}
W20260808 22:22:43.335013 23999 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:22:43.330079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.194:49123 (local address 127.9.176.254:44973)
0808 22:22:43.330210 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:43.330213 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:43.330332 (+ 119us) server_negotiation.cc:407] Connection header received
0808 22:22:43.330447 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:43.330448 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:43.330492 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:43.330546 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:43.330932 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:43.331444 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:43.332178 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:43.332385 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:43.332450 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:43.334916 (+ 2466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.176.194:49123: BlockingRecv error: recv got EOF from 127.9.176.194:49123 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 08 22:22:43 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.195@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:22:43 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.195@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:22:43.521023 24002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:43.515466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:44973 (local address 127.9.176.195:60389)
0808 22:22:43.515740 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:43.515756 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:43.515868 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:43.516142 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:43.516145 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:43.516374 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:43.516501 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:43.516508 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:43.517349 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:43.517352 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:43.517927 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:43.517935 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:43.518364 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:43.518367 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:43.518435 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:43.518954 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:43.518968 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:43.520849 (+ 1881us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.176.254:44973: Server oryx/127.9.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":106,"threads_started":1}
W20260808 22:22:43.521123 23999 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:22:43.515525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.195:60389 (local address 127.9.176.254:44973)
0808 22:22:43.515664 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:43.515669 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:43.515825 (+ 156us) server_negotiation.cc:407] Connection header received
0808 22:22:43.515955 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:43.515958 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:43.516016 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:43.516103 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:43.516606 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:43.517261 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:43.518014 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:43.518275 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:43.518347 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:43.521004 (+ 2657us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.176.195:60389: BlockingRecv error: recv got EOF from 127.9.176.195:60389 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 08 22:22:44 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.193@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:22:44 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.193@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:22:44.171010 24006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:44.166203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:44973 (local address 127.9.176.193:52015)
0808 22:22:44.166468 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:44.166481 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:44.166570 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:44.166789 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:44.166791 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:44.166971 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:44.167081 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:44.167086 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:44.167783 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:44.167785 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:44.168340 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:44.168346 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:44.168699 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:44.168701 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:44.168757 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:44.169217 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:44.169228 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:44.170898 (+ 1670us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.176.254:44973: Server oryx/127.9.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":85,"threads_started":1}
W20260808 22:22:44.171188 24007 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:22:44.166287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.193:52015 (local address 127.9.176.254:44973)
0808 22:22:44.166507 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:44.166510 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:44.166528 (+ 18us) server_negotiation.cc:407] Connection header received
0808 22:22:44.166646 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:44.166648 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:44.166690 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:44.166762 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:44.167177 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:44.167706 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:44.168416 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:44.168626 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:44.168691 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:44.171041 (+ 2350us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.176.193:52015: BlockingRecv error: recv got EOF from 127.9.176.193:52015 (error 108)
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":62,"threads_started":1}
Aug 08 22:22:44 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.194@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:22:44 dist-test-slave-ncp9 krb5kdc[22322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.176.194@KRBTEST.COM for oryx/127.9.176.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:22:44.341040 24008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:44.335817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:44973 (local address 127.9.176.194:60717)
0808 22:22:44.336063 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:44.336077 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:44.336169 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:44.336374 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:44.336376 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:44.336596 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:44.336695 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:44.336702 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:44.337482 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:44.337485 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:44.338068 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:44.338079 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:44.338518 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:44.338520 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:44.338580 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:44.339042 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:44.339055 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:44.340849 (+ 1794us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.176.254:44973: Server oryx/127.9.176.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":72,"threads_started":1}
W20260808 22:22:44.341125 24007 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:22:44.335902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.194:60717 (local address 127.9.176.254:44973)
0808 22:22:44.336040 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:44.336044 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:44.336134 (+ 90us) server_negotiation.cc:407] Connection header received
0808 22:22:44.336239 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:44.336242 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:44.336282 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:44.336351 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:44.336803 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:44.337364 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:44.338192 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:44.338421 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:44.338501 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:44.341022 (+ 2521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.176.194:60717: BlockingRecv error: recv got EOF from 127.9.176.194:60717 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 22:22:44.420058 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 23253
I20260808 22:22:44.431764 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 23484
I20260808 22:22:44.441246 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 23683
I20260808 22:22:44.459304 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 23141
2026-08-08T22:22:44Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17968 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-08T22:22:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:22:44Z Disabled control of system clock
I20260808 22:22:44.491279 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786227706282457-9923-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786227706282457-9923-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786227706282457-9923-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:39305
--webserver_interface=127.9.176.254
--webserver_port=0
--builtin_ntp_servers=127.9.176.212:40515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.176.254:39305
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260808 22:22:44.599385 24014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:22:44.599619 24014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:22:44.599651 24014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:22:44.603504 24014 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:22:44.603556 24014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:22:44.603569 24014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:22:44.603583 24014 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:22:44.603596 24014 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:22:44.608194 24014 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:40515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786227706282457-9923-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786227706282457-9923-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.176.254:39305
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:39305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.176.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24014
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786227706282457-9923-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:22:44.609141 24014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:22:44.610461 24014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 22:22:44.618816 24014 server_base.cc:1034] running on GCE node
W20260808 22:22:44.618849 24021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:44.618808 24024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:44.618808 24022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:22:44.619601 24014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:22:44.620633 24014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:22:44.621794 24014 hybrid_clock.cc:648] HybridClock initialized: now 1786227764621762 us; error 45 us; skew 500 ppm
I20260808 22:22:44.623715 24014 webserver.cc:460] Webserver started at http://127.9.176.254:44677/ using document root <none> and password file <none>
I20260808 22:22:44.624256 24014 fs_manager.cc:359] Metadata directory not provided
I20260808 22:22:44.624317 24014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:22:44.624500 24014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:22:44.626837 24014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786227706282457-9923-0/minicluster-data/master-0/data/instance:
uuid: "b5d48e63db4b4e58a2c103d9b463e648"
format_stamp: "Formatted at 2026-08-08 22:22:44 on dist-test-slave-ncp9"
server_key: "0c2be0bb8552e4e900f3ac3523955b1e"
server_key_iv: "7e6a0f99bb65d7bb42c16c4f897fb899"
server_key_version: "encryptionkey@0"
I20260808 22:22:44.627383 24014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786227706282457-9923-0/minicluster-data/master-0/wal/instance:
uuid: "b5d48e63db4b4e58a2c103d9b463e648"
format_stamp: "Formatted at 2026-08-08 22:22:44 on dist-test-slave-ncp9"
server_key: "0c2be0bb8552e4e900f3ac3523955b1e"
server_key_iv: "7e6a0f99bb65d7bb42c16c4f897fb899"
server_key_version: "encryptionkey@0"
I20260808 22:22:44.630906 24014 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260808 22:22:44.633301 24032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:44.634488 24014 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 22:22:44.634600 24014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786227706282457-9923-0/minicluster-data/master-0/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786227706282457-9923-0/minicluster-data/master-0/wal
uuid: "b5d48e63db4b4e58a2c103d9b463e648"
format_stamp: "Formatted at 2026-08-08 22:22:44 on dist-test-slave-ncp9"
server_key: "0c2be0bb8552e4e900f3ac3523955b1e"
server_key_iv: "7e6a0f99bb65d7bb42c16c4f897fb899"
server_key_version: "encryptionkey@0"
I20260808 22:22:44.634696 24014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786227706282457-9923-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786227706282457-9923-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786227706282457-9923-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:22:44.649507 24014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:22:44.650169 24014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:22:44.650336 24014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:22:44.656950 24014 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.254:39305
I20260808 22:22:44.656980 24112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.254:39305 every 8 connection(s)
I20260808 22:22:44.657858 24014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
I20260808 22:22:44.660590 24113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:22:44.666059 24113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5d48e63db4b4e58a2c103d9b463e648: Bootstrap starting.
I20260808 22:22:44.666867 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 24014
I20260808 22:22:44.666985 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786227706282457-9923-0/minicluster-data/master-0/wal/instance
I20260808 22:22:44.667285 9923 external_mini_cluster.cc:1428] Setting key 2601ca91af78cec32ad9861f09bf7134
I20260808 22:22:44.668880 24113 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5d48e63db4b4e58a2c103d9b463e648: Neither blocks nor log segments found. Creating new log.
I20260808 22:22:44.669621 24113 log.cc:824] T 00000000000000000000000000000000 P b5d48e63db4b4e58a2c103d9b463e648: Log is configured to *not* fsync() on all Append() calls
I20260808 22:22:44.671589 24113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5d48e63db4b4e58a2c103d9b463e648: No bootstrap required, opened a new log
I20260808 22:22:44.675576 24113 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5d48e63db4b4e58a2c103d9b463e648 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5d48e63db4b4e58a2c103d9b463e648" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 39305 } }
I20260808 22:22:44.675792 24113 raft_consensus.cc:374] T 00000000000000000000000000000000 P b5d48e63db4b4e58a2c103d9b463e648 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:22:44.675856 24113 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5d48e63db4b4e58a2c103d9b463e648 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5d48e63db4b4e58a2c103d9b463e648, State: Initialized, Role: FOLLOWER
I20260808 22:22:44.676265 24113 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5d48e63db4b4e58a2c103d9b463e648 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5d48e63db4b4e58a2c103d9b463e648" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 39305 } }
I20260808 22:22:44.676364 24116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:44.668410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36094 (local address 127.9.176.254:39305)
0808 22:22:44.669076 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:44.669092 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:44.669129 (+ 37us) server_negotiation.cc:407] Connection header received
0808 22:22:44.669751 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:44.669774 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:44.670162 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:44.670429 (+ 267us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:44.670787 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:44.671780 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:44.673005 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:44.673531 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:44.673687 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:44.674552 (+ 865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:44.674557 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:44.674569 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:22:44.674943 (+ 374us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:44.675001 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:22:44.675020 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:44.675097 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:44.675246 (+ 149us) server_negotiation.cc:299] Negotiation successful
0808 22:22:44.675538 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":111,"threads_started":1}
I20260808 22:22:44.676379 24113 raft_consensus.cc:388] T 00000000000000000000000000000000 P b5d48e63db4b4e58a2c103d9b463e648 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:22:44.676479 24113 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5d48e63db4b4e58a2c103d9b463e648 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:22:44.676570 24113 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5d48e63db4b4e58a2c103d9b463e648 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:22:44.677464 24113 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5d48e63db4b4e58a2c103d9b463e648 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5d48e63db4b4e58a2c103d9b463e648" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 39305 } }
I20260808 22:22:44.677809 24113 leader_election.cc:302] T 00000000000000000000000000000000 P b5d48e63db4b4e58a2c103d9b463e648 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5d48e63db4b4e58a2c103d9b463e648; no voters:
I20260808 22:22:44.678126 24113 leader_election.cc:288] T 00000000000000000000000000000000 P b5d48e63db4b4e58a2c103d9b463e648 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:22:44.678251 24118 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5d48e63db4b4e58a2c103d9b463e648 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:22:44.678480 24118 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5d48e63db4b4e58a2c103d9b463e648 [term 1 LEADER]: Becoming Leader. State: Replica: b5d48e63db4b4e58a2c103d9b463e648, State: Running, Role: LEADER
I20260808 22:22:44.678879 24118 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5d48e63db4b4e58a2c103d9b463e648 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5d48e63db4b4e58a2c103d9b463e648" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 39305 } }
I20260808 22:22:44.679219 24113 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5d48e63db4b4e58a2c103d9b463e648 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:22:44.679941 24118 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5d48e63db4b4e58a2c103d9b463e648 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5d48e63db4b4e58a2c103d9b463e648. Latest consensus state: current_term: 1 leader_uuid: "b5d48e63db4b4e58a2c103d9b463e648" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5d48e63db4b4e58a2c103d9b463e648" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 39305 } } }
I20260808 22:22:44.680696 24118 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5d48e63db4b4e58a2c103d9b463e648 [sys.catalog]: This master's current role is: LEADER
I20260808 22:22:44.680577 24121 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5d48e63db4b4e58a2c103d9b463e648 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b5d48e63db4b4e58a2c103d9b463e648" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5d48e63db4b4e58a2c103d9b463e648" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 39305 } } }
I20260808 22:22:44.680941 24121 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5d48e63db4b4e58a2c103d9b463e648 [sys.catalog]: This master's current role is: LEADER
W20260808 22:22:44.683413 24144 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b5d48e63db4b4e58a2c103d9b463e648: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 22:22:44.683477 24144 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 22:22:44.683550 24142 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:22:44.684223 24142 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:22:44.689550 24142 catalog_manager.cc:1347] Generated new cluster ID: c082486f65dd4f3dbee57b0d26070155
I20260808 22:22:44.689615 24142 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:22:44.722949 24142 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:22:44.724241 24142 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:22:44.730705 24142 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b5d48e63db4b4e58a2c103d9b463e648: Generated new TSK 0
I20260808 22:22:44.731518 24142 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 22:22:44.744220 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.193:0
--local_ip_for_outbound_sockets=127.9.176.193
--webserver_interface=127.9.176.193
--webserver_port=0
--tserver_master_addrs=127.9.176.254:39305
--builtin_ntp_servers=127.9.176.212:40515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260808 22:22:44.855679 24149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:22:44.855945 24149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:22:44.855981 24149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:22:44.859884 24149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:22:44.859933 24149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:22:44.859987 24149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.193
I20260808 22:22:44.864645 24149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:40515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.176.193
--webserver_port=0
--tserver_master_addrs=127.9.176.254:39305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.193
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:22:44.865696 24149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:22:44.867014 24149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:22:44.876494 24155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:44.876562 24161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:44.876495 24157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:22:44.876652 24149 server_base.cc:1034] running on GCE node
I20260808 22:22:44.877182 24149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:22:44.877771 24149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:22:44.878927 24149 hybrid_clock.cc:648] HybridClock initialized: now 1786227764878906 us; error 39 us; skew 500 ppm
I20260808 22:22:44.880745 24149 webserver.cc:460] Webserver started at http://127.9.176.193:45611/ using document root <none> and password file <none>
I20260808 22:22:44.881366 24149 fs_manager.cc:359] Metadata directory not provided
I20260808 22:22:44.881440 24149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:22:44.881655 24149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:22:44.883788 24149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-0/data/instance:
uuid: "61ddf141b30e43ed971c255a804b62c1"
format_stamp: "Formatted at 2026-08-08 22:22:44 on dist-test-slave-ncp9"
server_key: "e3fbe4859d738b8205f3bdec029a8f68"
server_key_iv: "38917e314130032adba56f9916ffd40f"
server_key_version: "encryptionkey@0"
I20260808 22:22:44.884343 24149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-0/wal/instance:
uuid: "61ddf141b30e43ed971c255a804b62c1"
format_stamp: "Formatted at 2026-08-08 22:22:44 on dist-test-slave-ncp9"
server_key: "e3fbe4859d738b8205f3bdec029a8f68"
server_key_iv: "38917e314130032adba56f9916ffd40f"
server_key_version: "encryptionkey@0"
I20260808 22:22:44.887995 24149 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260808 22:22:44.890509 24169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:44.891705 24149 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 22:22:44.891814 24149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-0/wal
uuid: "61ddf141b30e43ed971c255a804b62c1"
format_stamp: "Formatted at 2026-08-08 22:22:44 on dist-test-slave-ncp9"
server_key: "e3fbe4859d738b8205f3bdec029a8f68"
server_key_iv: "38917e314130032adba56f9916ffd40f"
server_key_version: "encryptionkey@0"
I20260808 22:22:44.891908 24149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:22:44.911986 24149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:22:44.912623 24149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:22:44.912801 24149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:22:44.913396 24149 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:22:44.914314 24149 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:22:44.914362 24149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:44.914414 24149 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:22:44.914451 24149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:44.922757 24149 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.193:37515
I20260808 22:22:44.922787 24366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.193:37515 every 8 connection(s)
I20260808 22:22:44.923626 24149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
I20260808 22:22:44.931062 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 24149
I20260808 22:22:44.931171 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-0/wal/instance
I20260808 22:22:44.931499 9923 external_mini_cluster.cc:1428] Setting key c9d1ceafb759a1a82fd997c628b0a542
I20260808 22:22:44.932317 24116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:44.925408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.193:49273 (local address 127.9.176.254:39305)
0808 22:22:44.925564 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:44.925568 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:44.926558 (+ 990us) server_negotiation.cc:407] Connection header received
0808 22:22:44.927870 (+ 1312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:44.927872 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:44.927917 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:44.927990 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:44.928872 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:44.929462 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:44.930525 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:44.930731 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:44.930816 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:44.931846 (+ 1030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:44.931850 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:44.931852 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:22:44.931867 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:44.931889 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:22:44.931898 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:44.931958 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:44.932133 (+ 175us) server_negotiation.cc:299] Negotiation successful
0808 22:22:44.932205 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 22:22:44.933359 24371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:44.925768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:39305 (local address 127.9.176.193:49273)
0808 22:22:44.926420 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:44.926461 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:44.927693 (+ 1232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:44.928147 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:44.928153 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:44.928235 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:44.928718 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:44.928731 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:44.929561 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:44.929564 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:44.930433 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:44.930443 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:44.930834 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:44.930837 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:44.930974 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:44.931584 (+ 610us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 22:22:44.931586 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:44.931634 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 22:22:44.931698 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:22:44.931986 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:22:44.931997 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:22:44.932003 (+ 6us) client_negotiation.cc:769] Sending connection context
0808 22:22:44.932124 (+ 121us) client_negotiation.cc:240] Negotiation successful
0808 22:22:44.932381 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":118,"threads_started":1}
I20260808 22:22:44.933447 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.194:0
--local_ip_for_outbound_sockets=127.9.176.194
--webserver_interface=127.9.176.194
--webserver_port=0
--tserver_master_addrs=127.9.176.254:39305
--builtin_ntp_servers=127.9.176.212:40515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260808 22:22:44.934484 24367 heartbeater.cc:347] Connected to a master server at 127.9.176.254:39305
I20260808 22:22:44.934696 24367 heartbeater.cc:464] Registering TS with master...
I20260808 22:22:44.935174 24367 heartbeater.cc:511] Master 127.9.176.254:39305 requested a full tablet report, sending...
I20260808 22:22:44.936661 24055 ts_manager.cc:194] Registered new tserver with Master: 61ddf141b30e43ed971c255a804b62c1 (127.9.176.193:37515)
I20260808 22:22:44.938177 24055 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.176.193:49273
W20260808 22:22:45.043449 24372 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:22:45.043694 24372 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:22:45.043728 24372 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:22:45.047580 24372 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:22:45.047622 24372 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:22:45.047670 24372 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.194
I20260808 22:22:45.052168 24372 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:40515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.176.194
--webserver_port=0
--tserver_master_addrs=127.9.176.254:39305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24372
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.194
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:22:45.053138 24372 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:22:45.054407 24372 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:22:45.063325 24383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:45.063294 24380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:45.063313 24379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:22:45.063464 24372 server_base.cc:1034] running on GCE node
I20260808 22:22:45.063927 24372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:22:45.064502 24372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:22:45.065651 24372 hybrid_clock.cc:648] HybridClock initialized: now 1786227765065639 us; error 25 us; skew 500 ppm
I20260808 22:22:45.067401 24372 webserver.cc:460] Webserver started at http://127.9.176.194:42915/ using document root <none> and password file <none>
I20260808 22:22:45.067915 24372 fs_manager.cc:359] Metadata directory not provided
I20260808 22:22:45.067991 24372 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:22:45.068189 24372 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:22:45.070165 24372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-1/data/instance:
uuid: "211fea921efa42229f511977ae2fa0bc"
format_stamp: "Formatted at 2026-08-08 22:22:45 on dist-test-slave-ncp9"
server_key: "4a0b96dfbfc70b5ce2fe92b2f7727d49"
server_key_iv: "8a8a37f92be3c26bd82178e63e0f101f"
server_key_version: "encryptionkey@0"
I20260808 22:22:45.070693 24372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-1/wal/instance:
uuid: "211fea921efa42229f511977ae2fa0bc"
format_stamp: "Formatted at 2026-08-08 22:22:45 on dist-test-slave-ncp9"
server_key: "4a0b96dfbfc70b5ce2fe92b2f7727d49"
server_key_iv: "8a8a37f92be3c26bd82178e63e0f101f"
server_key_version: "encryptionkey@0"
I20260808 22:22:45.074102 24372 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 22:22:45.076313 24392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:45.077471 24372 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 22:22:45.077587 24372 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-1/wal
uuid: "211fea921efa42229f511977ae2fa0bc"
format_stamp: "Formatted at 2026-08-08 22:22:45 on dist-test-slave-ncp9"
server_key: "4a0b96dfbfc70b5ce2fe92b2f7727d49"
server_key_iv: "8a8a37f92be3c26bd82178e63e0f101f"
server_key_version: "encryptionkey@0"
I20260808 22:22:45.077677 24372 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:22:45.093454 24372 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:22:45.094110 24372 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:22:45.094277 24372 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:22:45.094856 24372 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:22:45.095825 24372 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:22:45.095875 24372 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:45.095927 24372 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:22:45.095963 24372 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:45.103948 24372 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.194:44113
I20260808 22:22:45.103974 24567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.194:44113 every 8 connection(s)
I20260808 22:22:45.104789 24372 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
I20260808 22:22:45.110638 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 24372
I20260808 22:22:45.110743 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-1/wal/instance
I20260808 22:22:45.111068 9923 external_mini_cluster.cc:1428] Setting key 6021bcf595ed2176c8d4b898dd585763
I20260808 22:22:45.113063 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.195:0
--local_ip_for_outbound_sockets=127.9.176.195
--webserver_interface=127.9.176.195
--webserver_port=0
--tserver_master_addrs=127.9.176.254:39305
--builtin_ntp_servers=127.9.176.212:40515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260808 22:22:45.113778 24116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:45.106547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.194:40567 (local address 127.9.176.254:39305)
0808 22:22:45.106681 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:45.106684 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:45.107674 (+ 990us) server_negotiation.cc:407] Connection header received
0808 22:22:45.109032 (+ 1358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:45.109036 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:45.109096 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:45.109169 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:45.110045 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:45.110580 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:45.111850 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:45.112077 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:45.112165 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:45.113354 (+ 1189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:45.113357 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:45.113360 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:22:45.113372 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:45.113394 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:22:45.113404 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:45.113476 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:45.113585 (+ 109us) server_negotiation.cc:299] Negotiation successful
0808 22:22:45.113643 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 22:22:45.114562 24574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:45.106893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:39305 (local address 127.9.176.194:40567)
0808 22:22:45.107525 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:45.107568 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:45.108804 (+ 1236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:45.109348 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:45.109354 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:45.109435 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:45.109898 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:45.109910 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:45.110707 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:45.110710 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:45.111714 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:45.111727 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:45.112227 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:45.112231 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:45.112431 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:45.113086 (+ 655us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 22:22:45.113088 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:45.113128 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 22:22:45.113201 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:22:45.113476 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:22:45.113481 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:22:45.113485 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 22:22:45.113576 (+ 91us) client_negotiation.cc:240] Negotiation successful
0808 22:22:45.113787 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":116,"threads_started":1}
I20260808 22:22:45.115584 24568 heartbeater.cc:347] Connected to a master server at 127.9.176.254:39305
I20260808 22:22:45.115800 24568 heartbeater.cc:464] Registering TS with master...
I20260808 22:22:45.116263 24568 heartbeater.cc:511] Master 127.9.176.254:39305 requested a full tablet report, sending...
I20260808 22:22:45.117220 24055 ts_manager.cc:194] Registered new tserver with Master: 211fea921efa42229f511977ae2fa0bc (127.9.176.194:44113)
I20260808 22:22:45.117748 24055 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.176.194:40567
W20260808 22:22:45.223551 24575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:22:45.223819 24575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:22:45.223871 24575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:22:45.227892 24575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:22:45.227952 24575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:22:45.228026 24575 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.195
I20260808 22:22:45.232959 24575 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:40515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.176.195
--webserver_port=0
--tserver_master_addrs=127.9.176.254:39305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24575
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.195
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:22:45.234068 24575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:22:45.235419 24575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:22:45.244735 24587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:45.244763 24584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:22:45.244932 24575 server_base.cc:1034] running on GCE node
W20260808 22:22:45.245046 24583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:22:45.245532 24575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:22:45.246129 24575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:22:45.247298 24575 hybrid_clock.cc:648] HybridClock initialized: now 1786227765247281 us; error 26 us; skew 500 ppm
I20260808 22:22:45.249063 24575 webserver.cc:460] Webserver started at http://127.9.176.195:37935/ using document root <none> and password file <none>
I20260808 22:22:45.249603 24575 fs_manager.cc:359] Metadata directory not provided
I20260808 22:22:45.249670 24575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:22:45.249872 24575 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:22:45.251829 24575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-2/data/instance:
uuid: "b1ea2d570a5646f9ba3d6aa72d63b52b"
format_stamp: "Formatted at 2026-08-08 22:22:45 on dist-test-slave-ncp9"
server_key: "0830ac890d07fa21f7370c0b62651927"
server_key_iv: "562d1dad6f4e5234b7f3504d8757b15c"
server_key_version: "encryptionkey@0"
I20260808 22:22:45.252362 24575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-2/wal/instance:
uuid: "b1ea2d570a5646f9ba3d6aa72d63b52b"
format_stamp: "Formatted at 2026-08-08 22:22:45 on dist-test-slave-ncp9"
server_key: "0830ac890d07fa21f7370c0b62651927"
server_key_iv: "562d1dad6f4e5234b7f3504d8757b15c"
server_key_version: "encryptionkey@0"
I20260808 22:22:45.255841 24575 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260808 22:22:45.258200 24595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:45.259310 24575 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 22:22:45.259415 24575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-2/wal
uuid: "b1ea2d570a5646f9ba3d6aa72d63b52b"
format_stamp: "Formatted at 2026-08-08 22:22:45 on dist-test-slave-ncp9"
server_key: "0830ac890d07fa21f7370c0b62651927"
server_key_iv: "562d1dad6f4e5234b7f3504d8757b15c"
server_key_version: "encryptionkey@0"
I20260808 22:22:45.259516 24575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:22:45.279395 24575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:22:45.280105 24575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:22:45.280270 24575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:22:45.280864 24575 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:22:45.281876 24575 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:22:45.281926 24575 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:45.281966 24575 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:22:45.281994 24575 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:45.289908 24575 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.195:33049
I20260808 22:22:45.289963 24771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.195:33049 every 8 connection(s)
I20260808 22:22:45.290747 24575 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
I20260808 22:22:45.299444 24116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:45.292420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.195:40035 (local address 127.9.176.254:39305)
0808 22:22:45.292587 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:45.292592 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:45.293638 (+ 1046us) server_negotiation.cc:407] Connection header received
0808 22:22:45.295085 (+ 1447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:45.295088 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:45.295137 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:45.295214 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:45.296122 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:45.296650 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:45.297754 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:45.297967 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:45.298040 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:45.299062 (+ 1022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:45.299066 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:45.299068 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:22:45.299080 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:45.299104 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:22:45.299113 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:45.299176 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:45.299284 (+ 108us) server_negotiation.cc:299] Negotiation successful
0808 22:22:45.299333 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 22:22:45.299786 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 24575
I20260808 22:22:45.299877 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-2/wal/instance
I20260808 22:22:45.300145 9923 external_mini_cluster.cc:1428] Setting key 221a86a3272dd00bdd1d2621484f330d
I20260808 22:22:45.301084 24777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:45.292818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:39305 (local address 127.9.176.195:40035)
0808 22:22:45.293488 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:45.293536 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:45.294789 (+ 1253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:45.295360 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:45.295367 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:45.295448 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:45.295946 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:45.295959 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:45.296750 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:45.296753 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:45.297659 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:45.297668 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:45.298077 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:45.298080 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:45.298216 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:45.298799 (+ 583us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 22:22:45.298801 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:45.298841 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 22:22:45.298904 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:22:45.299181 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:22:45.299186 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:22:45.299189 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 22:22:45.299274 (+ 85us) client_negotiation.cc:240] Negotiation successful
0808 22:22:45.299480 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":125,"threads_started":1}
I20260808 22:22:45.302199 24775 heartbeater.cc:347] Connected to a master server at 127.9.176.254:39305
I20260808 22:22:45.302444 24775 heartbeater.cc:464] Registering TS with master...
I20260808 22:22:45.302945 24775 heartbeater.cc:511] Master 127.9.176.254:39305 requested a full tablet report, sending...
I20260808 22:22:45.303922 24054 ts_manager.cc:194] Registered new tserver with Master: b1ea2d570a5646f9ba3d6aa72d63b52b (127.9.176.195:33049)
I20260808 22:22:45.304471 24054 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.176.195:40035
I20260808 22:22:45.314177 9923 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260808 22:22:45.317165 24116 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:22:45.316198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36102 (local address 127.9.176.254:39305)
0808 22:22:45.316347 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:45.316353 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:45.316504 (+ 151us) server_negotiation.cc:407] Connection header received
0808 22:22:45.316620 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:45.316622 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:45.316668 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:45.316735 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:45.317036 (+ 301us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36102: BlockingRecv error: recv got EOF from 127.0.0.1:36102 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 22:22:45.317349 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 24149
I20260808 22:22:45.331326 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 24372
I20260808 22:22:45.347816 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 24575
I20260808 22:22:45.357693 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 24014
2026-08-08T22:22:45Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (900 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-08T22:22:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:22:45Z Disabled control of system clock
I20260808 22:22:45.391449 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786227706282457-9923-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786227706282457-9923-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786227706282457-9923-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:36705
--webserver_interface=127.9.176.254
--webserver_port=0
--builtin_ntp_servers=127.9.176.212:38901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.176.254:36705
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260808 22:22:45.500480 24796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:22:45.500720 24796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:22:45.500753 24796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:22:45.504627 24796 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:22:45.504684 24796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:22:45.504698 24796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:22:45.504714 24796 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:22:45.504733 24796 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:22:45.509382 24796 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:38901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786227706282457-9923-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786227706282457-9923-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.176.254:36705
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:36705
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.176.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24796
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786227706282457-9923-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:22:45.510367 24796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:22:45.511694 24796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 22:22:45.519850 24796 server_base.cc:1034] running on GCE node
W20260808 22:22:45.519778 24801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:45.519798 24807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:45.519793 24803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:22:45.520550 24796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:22:45.521581 24796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:22:45.522737 24796 hybrid_clock.cc:648] HybridClock initialized: now 1786227765522720 us; error 27 us; skew 500 ppm
I20260808 22:22:45.524540 24796 webserver.cc:460] Webserver started at http://127.9.176.254:41013/ using document root <none> and password file <none>
I20260808 22:22:45.525111 24796 fs_manager.cc:359] Metadata directory not provided
I20260808 22:22:45.525182 24796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:22:45.525378 24796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:22:45.527350 24796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786227706282457-9923-0/minicluster-data/master-0/data/instance:
uuid: "5e5fc968979341b9803a59b7c25702af"
format_stamp: "Formatted at 2026-08-08 22:22:45 on dist-test-slave-ncp9"
server_key: "4c78efe0742e8e5b908d1082e71b4f4e"
server_key_iv: "a8994f47c7e08ec88ff4d5e15ea11020"
server_key_version: "encryptionkey@0"
I20260808 22:22:45.527900 24796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786227706282457-9923-0/minicluster-data/master-0/wal/instance:
uuid: "5e5fc968979341b9803a59b7c25702af"
format_stamp: "Formatted at 2026-08-08 22:22:45 on dist-test-slave-ncp9"
server_key: "4c78efe0742e8e5b908d1082e71b4f4e"
server_key_iv: "a8994f47c7e08ec88ff4d5e15ea11020"
server_key_version: "encryptionkey@0"
I20260808 22:22:45.531380 24796 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260808 22:22:45.533699 24815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:45.534833 24796 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 22:22:45.534965 24796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786227706282457-9923-0/minicluster-data/master-0/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786227706282457-9923-0/minicluster-data/master-0/wal
uuid: "5e5fc968979341b9803a59b7c25702af"
format_stamp: "Formatted at 2026-08-08 22:22:45 on dist-test-slave-ncp9"
server_key: "4c78efe0742e8e5b908d1082e71b4f4e"
server_key_iv: "a8994f47c7e08ec88ff4d5e15ea11020"
server_key_version: "encryptionkey@0"
I20260808 22:22:45.535063 24796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786227706282457-9923-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786227706282457-9923-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786227706282457-9923-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:22:45.539541 24796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:22:45.540098 24796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:22:45.540253 24796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:22:45.546715 24796 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.254:36705
I20260808 22:22:45.546730 24912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.254:36705 every 8 connection(s)
I20260808 22:22:45.547572 24796 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
I20260808 22:22:45.550717 24913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:22:45.556097 24913 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e5fc968979341b9803a59b7c25702af: Bootstrap starting.
I20260808 22:22:45.556584 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 24796
I20260808 22:22:45.556685 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786227706282457-9923-0/minicluster-data/master-0/wal/instance
I20260808 22:22:45.557019 9923 external_mini_cluster.cc:1428] Setting key 6652c5ca5e04a471baa73aa8cd316564
I20260808 22:22:45.558701 24913 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e5fc968979341b9803a59b7c25702af: Neither blocks nor log segments found. Creating new log.
I20260808 22:22:45.559404 24913 log.cc:824] T 00000000000000000000000000000000 P 5e5fc968979341b9803a59b7c25702af: Log is configured to *not* fsync() on all Append() calls
I20260808 22:22:45.561599 24913 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e5fc968979341b9803a59b7c25702af: No bootstrap required, opened a new log
I20260808 22:22:45.562593 24922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:45.558311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35744 (local address 127.9.176.254:36705)
0808 22:22:45.558962 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:45.558980 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:45.559017 (+ 37us) server_negotiation.cc:407] Connection header received
0808 22:22:45.559728 (+ 711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:45.559754 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:45.560113 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:45.560451 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:45.561286 (+ 835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:45.561290 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:45.561301 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:22:45.561312 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:45.561369 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:22:45.561387 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:45.561512 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:45.561638 (+ 126us) server_negotiation.cc:299] Negotiation successful
0808 22:22:45.561880 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":122,"threads_started":1}
I20260808 22:22:45.565151 24913 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e5fc968979341b9803a59b7c25702af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e5fc968979341b9803a59b7c25702af" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 36705 } }
I20260808 22:22:45.565358 24913 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e5fc968979341b9803a59b7c25702af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:22:45.565414 24913 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e5fc968979341b9803a59b7c25702af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e5fc968979341b9803a59b7c25702af, State: Initialized, Role: FOLLOWER
I20260808 22:22:45.565788 24913 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e5fc968979341b9803a59b7c25702af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e5fc968979341b9803a59b7c25702af" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 36705 } }
I20260808 22:22:45.565907 24913 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e5fc968979341b9803a59b7c25702af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:22:45.565965 24913 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e5fc968979341b9803a59b7c25702af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:22:45.566066 24913 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e5fc968979341b9803a59b7c25702af [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:22:45.566895 24913 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e5fc968979341b9803a59b7c25702af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e5fc968979341b9803a59b7c25702af" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 36705 } }
I20260808 22:22:45.567203 24913 leader_election.cc:302] T 00000000000000000000000000000000 P 5e5fc968979341b9803a59b7c25702af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e5fc968979341b9803a59b7c25702af; no voters:
I20260808 22:22:45.567471 24913 leader_election.cc:288] T 00000000000000000000000000000000 P 5e5fc968979341b9803a59b7c25702af [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:22:45.567595 24924 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e5fc968979341b9803a59b7c25702af [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:22:45.567821 24924 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e5fc968979341b9803a59b7c25702af [term 1 LEADER]: Becoming Leader. State: Replica: 5e5fc968979341b9803a59b7c25702af, State: Running, Role: LEADER
I20260808 22:22:45.568140 24924 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e5fc968979341b9803a59b7c25702af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e5fc968979341b9803a59b7c25702af" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 36705 } }
I20260808 22:22:45.568594 24913 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e5fc968979341b9803a59b7c25702af [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:22:45.569667 24925 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e5fc968979341b9803a59b7c25702af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e5fc968979341b9803a59b7c25702af. Latest consensus state: current_term: 1 leader_uuid: "5e5fc968979341b9803a59b7c25702af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e5fc968979341b9803a59b7c25702af" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 36705 } } }
I20260808 22:22:45.569810 24925 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e5fc968979341b9803a59b7c25702af [sys.catalog]: This master's current role is: LEADER
I20260808 22:22:45.570185 24936 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:22:45.570472 24927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e5fc968979341b9803a59b7c25702af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5e5fc968979341b9803a59b7c25702af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e5fc968979341b9803a59b7c25702af" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 36705 } } }
I20260808 22:22:45.570611 24927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e5fc968979341b9803a59b7c25702af [sys.catalog]: This master's current role is: LEADER
I20260808 22:22:45.573711 24936 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:22:45.578369 24936 catalog_manager.cc:1347] Generated new cluster ID: 1e9305744dcf49fa974adf7fb94a2734
I20260808 22:22:45.578433 24936 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:22:45.610137 24936 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:22:45.610450 24936 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:22:45.618388 24936 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5e5fc968979341b9803a59b7c25702af: Generated new TSK 0
I20260808 22:22:45.619040 24936 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 22:22:45.626993 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.193:0
--local_ip_for_outbound_sockets=127.9.176.193
--webserver_interface=127.9.176.193
--webserver_port=0
--tserver_master_addrs=127.9.176.254:36705
--builtin_ntp_servers=127.9.176.212:38901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260808 22:22:45.735769 24954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:22:45.736003 24954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:22:45.736039 24954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:22:45.739959 24954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:22:45.740007 24954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:22:45.740065 24954 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.193
I20260808 22:22:45.744587 24954 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:38901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.176.193
--webserver_port=0
--tserver_master_addrs=127.9.176.254:36705
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24954
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.193
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:22:45.745566 24954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:22:45.746866 24954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 22:22:45.756049 24954 server_base.cc:1034] running on GCE node
W20260808 22:22:45.756068 24963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:45.756109 24968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:45.756054 24964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:22:45.756660 24954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:22:45.757265 24954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:22:45.758446 24954 hybrid_clock.cc:648] HybridClock initialized: now 1786227765758431 us; error 29 us; skew 500 ppm
I20260808 22:22:45.760201 24954 webserver.cc:460] Webserver started at http://127.9.176.193:44645/ using document root <none> and password file <none>
I20260808 22:22:45.760722 24954 fs_manager.cc:359] Metadata directory not provided
I20260808 22:22:45.760787 24954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:22:45.761011 24954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:22:45.762979 24954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-0/data/instance:
uuid: "688d5c6aa2be4204a4d7469ee65985d1"
format_stamp: "Formatted at 2026-08-08 22:22:45 on dist-test-slave-ncp9"
server_key: "7bb016676454521646c18db0fffb4f54"
server_key_iv: "838c66c09f8fd4a0937697744ddf0fad"
server_key_version: "encryptionkey@0"
I20260808 22:22:45.763538 24954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-0/wal/instance:
uuid: "688d5c6aa2be4204a4d7469ee65985d1"
format_stamp: "Formatted at 2026-08-08 22:22:45 on dist-test-slave-ncp9"
server_key: "7bb016676454521646c18db0fffb4f54"
server_key_iv: "838c66c09f8fd4a0937697744ddf0fad"
server_key_version: "encryptionkey@0"
I20260808 22:22:45.767001 24954 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 22:22:45.769424 24982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:45.770670 24954 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 22:22:45.770777 24954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-0/wal
uuid: "688d5c6aa2be4204a4d7469ee65985d1"
format_stamp: "Formatted at 2026-08-08 22:22:45 on dist-test-slave-ncp9"
server_key: "7bb016676454521646c18db0fffb4f54"
server_key_iv: "838c66c09f8fd4a0937697744ddf0fad"
server_key_version: "encryptionkey@0"
I20260808 22:22:45.770869 24954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:22:45.775230 24954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:22:45.775820 24954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:22:45.775971 24954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:22:45.776463 24954 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:22:45.777344 24954 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:22:45.777398 24954 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:45.777448 24954 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:22:45.777493 24954 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:45.785398 24954 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.193:39807
I20260808 22:22:45.785447 25169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.193:39807 every 8 connection(s)
I20260808 22:22:45.786230 24954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
I20260808 22:22:45.792109 24922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:45.788032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.193:44089 (local address 127.9.176.254:36705)
0808 22:22:45.788190 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:45.788196 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:45.789269 (+ 1073us) server_negotiation.cc:407] Connection header received
0808 22:22:45.790517 (+ 1248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:45.790520 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:45.790561 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:45.790645 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:45.791692 (+ 1047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:45.791695 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:45.791697 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:22:45.791711 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:45.791734 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:22:45.791743 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:45.791823 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:45.791965 (+ 142us) server_negotiation.cc:299] Negotiation successful
0808 22:22:45.792002 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 22:22:45.792979 25175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:45.788448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:36705 (local address 127.9.176.193:44089)
0808 22:22:45.789129 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:45.789170 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:45.790320 (+ 1150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:45.790788 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:45.790794 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:45.790878 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:45.791384 (+ 506us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 22:22:45.791386 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:45.791438 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 22:22:45.791528 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:22:45.791826 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:22:45.791834 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:22:45.791840 (+ 6us) client_negotiation.cc:769] Sending connection context
0808 22:22:45.792007 (+ 167us) client_negotiation.cc:240] Negotiation successful
0808 22:22:45.792246 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":119,"threads_started":1}
I20260808 22:22:45.793669 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 24954
I20260808 22:22:45.793757 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-0/wal/instance
I20260808 22:22:45.794042 9923 external_mini_cluster.cc:1428] Setting key 519a3c4d4e7e783c6ceba79ad5d1657e
I20260808 22:22:45.794119 25172 heartbeater.cc:347] Connected to a master server at 127.9.176.254:36705
I20260808 22:22:45.794335 25172 heartbeater.cc:464] Registering TS with master...
I20260808 22:22:45.794761 25172 heartbeater.cc:511] Master 127.9.176.254:36705 requested a full tablet report, sending...
I20260808 22:22:45.795882 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.194:0
--local_ip_for_outbound_sockets=127.9.176.194
--webserver_interface=127.9.176.194
--webserver_port=0
--tserver_master_addrs=127.9.176.254:36705
--builtin_ntp_servers=127.9.176.212:38901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260808 22:22:45.796231 24847 ts_manager.cc:194] Registered new tserver with Master: 688d5c6aa2be4204a4d7469ee65985d1 (127.9.176.193:39807)
W20260808 22:22:45.903813 25176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:22:45.904054 25176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:22:45.904093 25176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:22:45.908041 25176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:22:45.908088 25176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:22:45.908149 25176 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.194
I20260808 22:22:45.912719 25176 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:38901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.176.194
--webserver_port=0
--tserver_master_addrs=127.9.176.254:36705
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25176
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.194
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:22:45.913674 25176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:22:45.914934 25176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:22:45.923985 25187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:22:45.924033 25176 server_base.cc:1034] running on GCE node
W20260808 22:22:45.923997 25190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:45.924000 25186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:22:45.924625 25176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:22:45.925254 25176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:22:45.926396 25176 hybrid_clock.cc:648] HybridClock initialized: now 1786227765926367 us; error 42 us; skew 500 ppm
I20260808 22:22:45.928112 25176 webserver.cc:460] Webserver started at http://127.9.176.194:35121/ using document root <none> and password file <none>
I20260808 22:22:45.928629 25176 fs_manager.cc:359] Metadata directory not provided
I20260808 22:22:45.928691 25176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:22:45.928875 25176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:22:45.930846 25176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-1/data/instance:
uuid: "f408a211eee0413798df3a5b2d7fd681"
format_stamp: "Formatted at 2026-08-08 22:22:45 on dist-test-slave-ncp9"
server_key: "039bd0e17883a31125601d9201a32f1a"
server_key_iv: "2a5f3af3bac9821205f4a93a234ebaf7"
server_key_version: "encryptionkey@0"
I20260808 22:22:45.931396 25176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-1/wal/instance:
uuid: "f408a211eee0413798df3a5b2d7fd681"
format_stamp: "Formatted at 2026-08-08 22:22:45 on dist-test-slave-ncp9"
server_key: "039bd0e17883a31125601d9201a32f1a"
server_key_iv: "2a5f3af3bac9821205f4a93a234ebaf7"
server_key_version: "encryptionkey@0"
I20260808 22:22:45.934873 25176 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 22:22:45.937206 25197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:45.938393 25176 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 22:22:45.938495 25176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-1/wal
uuid: "f408a211eee0413798df3a5b2d7fd681"
format_stamp: "Formatted at 2026-08-08 22:22:45 on dist-test-slave-ncp9"
server_key: "039bd0e17883a31125601d9201a32f1a"
server_key_iv: "2a5f3af3bac9821205f4a93a234ebaf7"
server_key_version: "encryptionkey@0"
I20260808 22:22:45.938597 25176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:22:45.942978 25176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:22:45.943528 25176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:22:45.943691 25176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:22:45.944190 25176 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:22:45.945057 25176 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:22:45.945104 25176 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:45.945152 25176 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:22:45.945181 25176 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:45.953603 25176 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.194:39445
I20260808 22:22:45.953635 25391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.194:39445 every 8 connection(s)
I20260808 22:22:45.954430 25176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
I20260808 22:22:45.960191 24922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:45.956084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.194:34135 (local address 127.9.176.254:36705)
0808 22:22:45.956212 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:45.956216 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:45.957287 (+ 1071us) server_negotiation.cc:407] Connection header received
0808 22:22:45.958569 (+ 1282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:45.958573 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:45.958619 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:45.958691 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:45.959745 (+ 1054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:45.959748 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:45.959751 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:22:45.959764 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:45.959785 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:22:45.959793 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:45.959871 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:45.960010 (+ 139us) server_negotiation.cc:299] Negotiation successful
0808 22:22:45.960056 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260808 22:22:45.961002 25395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:45.956412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:36705 (local address 127.9.176.194:34135)
0808 22:22:45.957136 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:45.957176 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:45.958350 (+ 1174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:45.958857 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:45.958864 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:45.958957 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:45.959481 (+ 524us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 22:22:45.959482 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:45.959523 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 22:22:45.959590 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:22:45.959862 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:22:45.959870 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:22:45.959877 (+ 7us) client_negotiation.cc:769] Sending connection context
0808 22:22:45.960039 (+ 162us) client_negotiation.cc:240] Negotiation successful
0808 22:22:45.960264 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":111,"threads_started":1}
I20260808 22:22:45.961833 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 25176
I20260808 22:22:45.961946 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-1/wal/instance
I20260808 22:22:45.962054 25392 heartbeater.cc:347] Connected to a master server at 127.9.176.254:36705
I20260808 22:22:45.962246 9923 external_mini_cluster.cc:1428] Setting key 29b1facb52a9893b0f4a37b82b890530
I20260808 22:22:45.962280 25392 heartbeater.cc:464] Registering TS with master...
I20260808 22:22:45.962646 25392 heartbeater.cc:511] Master 127.9.176.254:36705 requested a full tablet report, sending...
I20260808 22:22:45.963589 24847 ts_manager.cc:194] Registered new tserver with Master: f408a211eee0413798df3a5b2d7fd681 (127.9.176.194:39445)
I20260808 22:22:45.964032 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.195:0
--local_ip_for_outbound_sockets=127.9.176.195
--webserver_interface=127.9.176.195
--webserver_port=0
--tserver_master_addrs=127.9.176.254:36705
--builtin_ntp_servers=127.9.176.212:38901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260808 22:22:46.073061 25397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:22:46.073287 25397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:22:46.073325 25397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:22:46.077339 25397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:22:46.077390 25397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:22:46.077442 25397 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.195
I20260808 22:22:46.082067 25397 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:38901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.176.195
--webserver_port=0
--tserver_master_addrs=127.9.176.254:36705
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25397
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.195
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:22:46.083062 25397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:22:46.084385 25397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 22:22:46.093832 25397 server_base.cc:1034] running on GCE node
W20260808 22:22:46.093791 25409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:46.093804 25412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:46.093789 25408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:22:46.094439 25397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:22:46.095022 25397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:22:46.096174 25397 hybrid_clock.cc:648] HybridClock initialized: now 1786227766096157 us; error 29 us; skew 500 ppm
I20260808 22:22:46.097929 25397 webserver.cc:460] Webserver started at http://127.9.176.195:39487/ using document root <none> and password file <none>
I20260808 22:22:46.098476 25397 fs_manager.cc:359] Metadata directory not provided
I20260808 22:22:46.098537 25397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:22:46.098737 25397 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:22:46.100800 25397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-2/data/instance:
uuid: "6361e8aee453428b91f185b6c87f056b"
format_stamp: "Formatted at 2026-08-08 22:22:46 on dist-test-slave-ncp9"
server_key: "82fcff66df050706c4799334ee8c87f3"
server_key_iv: "b45843e0a9a7b65b2dca681fdc0ae069"
server_key_version: "encryptionkey@0"
I20260808 22:22:46.101393 25397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-2/wal/instance:
uuid: "6361e8aee453428b91f185b6c87f056b"
format_stamp: "Formatted at 2026-08-08 22:22:46 on dist-test-slave-ncp9"
server_key: "82fcff66df050706c4799334ee8c87f3"
server_key_iv: "b45843e0a9a7b65b2dca681fdc0ae069"
server_key_version: "encryptionkey@0"
I20260808 22:22:46.104877 25397 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260808 22:22:46.107307 25421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:46.108487 25397 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260808 22:22:46.108583 25397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-2/wal
uuid: "6361e8aee453428b91f185b6c87f056b"
format_stamp: "Formatted at 2026-08-08 22:22:46 on dist-test-slave-ncp9"
server_key: "82fcff66df050706c4799334ee8c87f3"
server_key_iv: "b45843e0a9a7b65b2dca681fdc0ae069"
server_key_version: "encryptionkey@0"
I20260808 22:22:46.108666 25397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:22:46.113056 25397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:22:46.113584 25397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:22:46.113716 25397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:22:46.114180 25397 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:22:46.115018 25397 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:22:46.115072 25397 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:46.115108 25397 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:22:46.115127 25397 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:46.124235 25397 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.195:41927
I20260808 22:22:46.124415 25627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.195:41927 every 8 connection(s)
I20260808 22:22:46.125126 25397 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
I20260808 22:22:46.130322 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 25397
I20260808 22:22:46.130414 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786227706282457-9923-0/minicluster-data/ts-2/wal/instance
I20260808 22:22:46.130679 9923 external_mini_cluster.cc:1428] Setting key a8d6d54cf52f2d2cee53b91ec4a6add9
I20260808 22:22:46.130713 24922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:46.126760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.195:40671 (local address 127.9.176.254:36705)
0808 22:22:46.126866 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:46.126869 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:46.127861 (+ 992us) server_negotiation.cc:407] Connection header received
0808 22:22:46.129165 (+ 1304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:46.129168 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:46.129204 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:46.129274 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:46.130240 (+ 966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:46.130244 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:46.130247 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:22:46.130260 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:46.130281 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:22:46.130289 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:46.130364 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:46.130540 (+ 176us) server_negotiation.cc:299] Negotiation successful
0808 22:22:46.130585 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260808 22:22:46.131593 25633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:46.127116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:36705 (local address 127.9.176.195:40671)
0808 22:22:46.127726 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:46.127763 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:46.128989 (+ 1226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:46.129392 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:46.129398 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:46.129485 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:46.129988 (+ 503us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 22:22:46.129990 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:46.130027 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 22:22:46.130090 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:22:46.130357 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:22:46.130365 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:22:46.130370 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 22:22:46.130571 (+ 201us) client_negotiation.cc:240] Negotiation successful
0808 22:22:46.130791 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":104,"threads_started":1}
I20260808 22:22:46.132563 25628 heartbeater.cc:347] Connected to a master server at 127.9.176.254:36705
I20260808 22:22:46.132764 25628 heartbeater.cc:464] Registering TS with master...
I20260808 22:22:46.133155 25628 heartbeater.cc:511] Master 127.9.176.254:36705 requested a full tablet report, sending...
I20260808 22:22:46.134056 24847 ts_manager.cc:194] Registered new tserver with Master: 6361e8aee453428b91f185b6c87f056b (127.9.176.195:41927)
I20260808 22:22:46.144177 9923 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260808 22:22:46.146903 24922 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:22:46.146046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35746 (local address 127.9.176.254:36705)
0808 22:22:46.146179 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:46.146184 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:46.146334 (+ 150us) server_negotiation.cc:407] Connection header received
0808 22:22:46.146451 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:46.146454 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:46.146492 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:46.146560 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:46.146807 (+ 247us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35746: BlockingRecv error: recv got EOF from 127.0.0.1:35746 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 22:22:46.147150 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 24954
I20260808 22:22:46.171607 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 25176
I20260808 22:22:46.182832 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 25397
I20260808 22:22:46.189450 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 24796
2026-08-08T22:22:46Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (830 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 22:22:46 dist-test-slave-ncp9 krb5kdc[25644](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 22:22:46 dist-test-slave-ncp9 krb5kdc[25644](info): set up 2 sockets
Aug 08 22:22:46 dist-test-slave-ncp9 krb5kdc[25644](info): commencing operation
krb5kdc: starting...
W20260808 22:22:47.325093 9923 mini_kdc.cc:121] Time spent starting KDC: real 1.111s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/krb5kdc/test-user.keytab.
Aug 08 22:22:47 dist-test-slave-ncp9 krb5kdc[25644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227767, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T22:22:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:22:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.176.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.176.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.176.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab.
Entry for principal kudu/127.9.176.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab.
I20260808 22:22:47.441694 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:38921
--webserver_interface=127.9.176.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab
--principal=kudu/127.9.176.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:44439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.176.254:38921
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:22:47.550807 25663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:22:47.551038 25663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:22:47.551083 25663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:22:47.554941 25663 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:22:47.554989 25663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:22:47.555007 25663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:22:47.555022 25663 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:22:47.555034 25663 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:22:47.560060 25663 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:44439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.176.254:38921
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab
--principal=kudu/127.9.176.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:38921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.176.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25663
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:22:47.560987 25663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:22:47.561900 25663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:22:47.569921 25670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:47.569979 25673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:22:47.569957 25663 server_base.cc:1034] running on GCE node
W20260808 22:22:47.569983 25671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:22:47.570796 25663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:22:47.571781 25663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:22:47.572945 25663 hybrid_clock.cc:648] HybridClock initialized: now 1786227767572928 us; error 27 us; skew 500 ppm
Aug 08 22:22:47 dist-test-slave-ncp9 krb5kdc[25644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227767, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:22:47.575548 25663 init.cc:377] Logged in from keytab as kudu/127.9.176.254@KRBTEST.COM (short username kudu)
I20260808 22:22:47.576319 25663 webserver.cc:460] Webserver started at http://127.9.176.254:40799/ using document root <none> and password file <none>
I20260808 22:22:47.576798 25663 fs_manager.cc:359] Metadata directory not provided
I20260808 22:22:47.576851 25663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:22:47.577072 25663 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:22:47.578965 25663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/minicluster-data/master-0/data/instance:
uuid: "9a16ed4c823642c79c71704dd9471cdb"
format_stamp: "Formatted at 2026-08-08 22:22:47 on dist-test-slave-ncp9"
server_key: "4f4304fa9cb82108d7c78d3d38a6d291"
server_key_iv: "1239391e7460c5f94d5660e9350f87fb"
server_key_version: "encryptionkey@0"
I20260808 22:22:47.579465 25663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/minicluster-data/master-0/wal/instance:
uuid: "9a16ed4c823642c79c71704dd9471cdb"
format_stamp: "Formatted at 2026-08-08 22:22:47 on dist-test-slave-ncp9"
server_key: "4f4304fa9cb82108d7c78d3d38a6d291"
server_key_iv: "1239391e7460c5f94d5660e9350f87fb"
server_key_version: "encryptionkey@0"
I20260808 22:22:47.582911 25663 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 22:22:47.585126 25687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:47.586244 25663 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 22:22:47.586369 25663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/minicluster-data/master-0/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/minicluster-data/master-0/wal
uuid: "9a16ed4c823642c79c71704dd9471cdb"
format_stamp: "Formatted at 2026-08-08 22:22:47 on dist-test-slave-ncp9"
server_key: "4f4304fa9cb82108d7c78d3d38a6d291"
server_key_iv: "1239391e7460c5f94d5660e9350f87fb"
server_key_version: "encryptionkey@0"
I20260808 22:22:47.586457 25663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:22:47.603470 25663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:22:47.606308 25663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:22:47.606474 25663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:22:47.612962 25663 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.254:38921
I20260808 22:22:47.613251 25769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.254:38921 every 8 connection(s)
I20260808 22:22:47.613965 25663 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
I20260808 22:22:47.616688 25770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:22:47.617645 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 25663
I20260808 22:22:47.617764 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/minicluster-data/master-0/wal/instance
I20260808 22:22:47.618083 9923 external_mini_cluster.cc:1428] Setting key 65692ed0b6920b22fdeda717128cf8bb
I20260808 22:22:47.623505 25770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a16ed4c823642c79c71704dd9471cdb: Bootstrap starting.
I20260808 22:22:47.625977 25770 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a16ed4c823642c79c71704dd9471cdb: Neither blocks nor log segments found. Creating new log.
I20260808 22:22:47.626684 25770 log.cc:824] T 00000000000000000000000000000000 P 9a16ed4c823642c79c71704dd9471cdb: Log is configured to *not* fsync() on all Append() calls
Aug 08 22:22:47 dist-test-slave-ncp9 krb5kdc[25644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227767, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
I20260808 22:22:47.628732 25770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a16ed4c823642c79c71704dd9471cdb: No bootstrap required, opened a new log
I20260808 22:22:47.632118 25774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:47.619413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50132 (local address 127.9.176.254:38921)
0808 22:22:47.619823 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:47.619843 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:47.619887 (+ 44us) server_negotiation.cc:407] Connection header received
0808 22:22:47.620690 (+ 803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:47.620715 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:47.621169 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:47.621532 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:47.622193 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:47.623666 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:47.624490 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:47.625213 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:47.625362 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:47.627811 (+ 2449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:47.627816 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:47.627829 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:22:47.627853 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:47.629481 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:47.629942 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:47.629946 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:47.629950 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:47.630064 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:47.630319 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:47.630320 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:47.630321 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:47.630690 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:47.630822 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:47.631124 (+ 302us) server_negotiation.cc:299] Negotiation successful
0808 22:22:47.631378 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":113,"threads_started":1}
I20260808 22:22:47.632555 25770 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a16ed4c823642c79c71704dd9471cdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a16ed4c823642c79c71704dd9471cdb" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 38921 } }
I20260808 22:22:47.632782 25770 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a16ed4c823642c79c71704dd9471cdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:22:47.632840 25770 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a16ed4c823642c79c71704dd9471cdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a16ed4c823642c79c71704dd9471cdb, State: Initialized, Role: FOLLOWER
I20260808 22:22:47.633289 25770 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a16ed4c823642c79c71704dd9471cdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a16ed4c823642c79c71704dd9471cdb" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 38921 } }
I20260808 22:22:47.633430 25770 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a16ed4c823642c79c71704dd9471cdb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:22:47.633495 25770 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a16ed4c823642c79c71704dd9471cdb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:22:47.633592 25770 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a16ed4c823642c79c71704dd9471cdb [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:22:47.634524 25770 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a16ed4c823642c79c71704dd9471cdb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a16ed4c823642c79c71704dd9471cdb" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 38921 } }
I20260808 22:22:47.634832 25770 leader_election.cc:302] T 00000000000000000000000000000000 P 9a16ed4c823642c79c71704dd9471cdb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a16ed4c823642c79c71704dd9471cdb; no voters:
I20260808 22:22:47.635102 25770 leader_election.cc:288] T 00000000000000000000000000000000 P 9a16ed4c823642c79c71704dd9471cdb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:22:47.635210 25776 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a16ed4c823642c79c71704dd9471cdb [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:22:47.635402 25776 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a16ed4c823642c79c71704dd9471cdb [term 1 LEADER]: Becoming Leader. State: Replica: 9a16ed4c823642c79c71704dd9471cdb, State: Running, Role: LEADER
I20260808 22:22:47.635702 25776 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a16ed4c823642c79c71704dd9471cdb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a16ed4c823642c79c71704dd9471cdb" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 38921 } }
I20260808 22:22:47.636204 25770 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a16ed4c823642c79c71704dd9471cdb [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:22:47.636780 25776 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a16ed4c823642c79c71704dd9471cdb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a16ed4c823642c79c71704dd9471cdb. Latest consensus state: current_term: 1 leader_uuid: "9a16ed4c823642c79c71704dd9471cdb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a16ed4c823642c79c71704dd9471cdb" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 38921 } } }
I20260808 22:22:47.636996 25776 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a16ed4c823642c79c71704dd9471cdb [sys.catalog]: This master's current role is: LEADER
I20260808 22:22:47.637316 25781 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a16ed4c823642c79c71704dd9471cdb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a16ed4c823642c79c71704dd9471cdb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a16ed4c823642c79c71704dd9471cdb" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 38921 } } }
I20260808 22:22:47.637481 25781 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a16ed4c823642c79c71704dd9471cdb [sys.catalog]: This master's current role is: LEADER
I20260808 22:22:47.637478 25789 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:22:47.640413 25789 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:22:47.646783 25789 catalog_manager.cc:1347] Generated new cluster ID: deb9b5b968834a6dae7ef4349b6b11ca
I20260808 22:22:47.646842 25789 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:22:47.659161 25789 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:22:47.660077 25789 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:22:47.665472 25789 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a16ed4c823642c79c71704dd9471cdb: Generated new TSK 0
I20260808 22:22:47.666265 25789 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.176.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.176.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.176.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.193.keytab.
Entry for principal kudu/127.9.176.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.193.keytab.
I20260808 22:22:47.702226 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.193:0
--local_ip_for_outbound_sockets=127.9.176.193
--webserver_interface=127.9.176.193
--webserver_port=0
--tserver_master_addrs=127.9.176.254:38921
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.193.keytab
--principal=kudu/127.9.176.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:44439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:22:47.812835 25817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:22:47.813088 25817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:22:47.813128 25817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:22:47.817011 25817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:22:47.817060 25817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:22:47.817113 25817 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.193
I20260808 22:22:47.822274 25817 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:44439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.193.keytab
--principal=kudu/127.9.176.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.176.193
--webserver_port=0
--tserver_master_addrs=127.9.176.254:38921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25817
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.193
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:22:47.823292 25817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:22:47.824261 25817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:22:47.833582 25825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:47.833596 25827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:22:47.833668 25817 server_base.cc:1034] running on GCE node
W20260808 22:22:47.833601 25830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:22:47.834206 25817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:22:47.834832 25817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:22:47.835992 25817 hybrid_clock.cc:648] HybridClock initialized: now 1786227767835975 us; error 27 us; skew 500 ppm
Aug 08 22:22:47 dist-test-slave-ncp9 krb5kdc[25644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227767, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:22:47.838753 25817 init.cc:377] Logged in from keytab as kudu/127.9.176.193@KRBTEST.COM (short username kudu)
I20260808 22:22:47.839576 25817 webserver.cc:460] Webserver started at http://127.9.176.193:41541/ using document root <none> and password file <none>
I20260808 22:22:47.840098 25817 fs_manager.cc:359] Metadata directory not provided
I20260808 22:22:47.840165 25817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:22:47.840363 25817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:22:47.842358 25817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/minicluster-data/ts-0/data/instance:
uuid: "39a3df6787e74130b0aef5977074117c"
format_stamp: "Formatted at 2026-08-08 22:22:47 on dist-test-slave-ncp9"
server_key: "f64b47fb121490a2160525f83603b639"
server_key_iv: "a72cd6b7fd7f488c1b9bfe56a8ecf7da"
server_key_version: "encryptionkey@0"
I20260808 22:22:47.842901 25817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/minicluster-data/ts-0/wal/instance:
uuid: "39a3df6787e74130b0aef5977074117c"
format_stamp: "Formatted at 2026-08-08 22:22:47 on dist-test-slave-ncp9"
server_key: "f64b47fb121490a2160525f83603b639"
server_key_iv: "a72cd6b7fd7f488c1b9bfe56a8ecf7da"
server_key_version: "encryptionkey@0"
I20260808 22:22:47.846402 25817 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260808 22:22:47.848675 25842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:47.849851 25817 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 22:22:47.849961 25817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/minicluster-data/ts-0/wal
uuid: "39a3df6787e74130b0aef5977074117c"
format_stamp: "Formatted at 2026-08-08 22:22:47 on dist-test-slave-ncp9"
server_key: "f64b47fb121490a2160525f83603b639"
server_key_iv: "a72cd6b7fd7f488c1b9bfe56a8ecf7da"
server_key_version: "encryptionkey@0"
I20260808 22:22:47.850051 25817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:22:47.884197 25817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:22:47.887496 25817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:22:47.887676 25817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:22:47.888327 25817 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:22:47.889228 25817 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:22:47.889276 25817 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:47.889329 25817 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:22:47.889357 25817 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:47.898280 25817 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.193:43669
I20260808 22:22:47.898306 26022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.193:43669 every 8 connection(s)
I20260808 22:22:47.899196 25817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
Aug 08 22:22:47 dist-test-slave-ncp9 krb5kdc[25644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227767, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.193@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
I20260808 22:22:47.909400 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 25817
I20260808 22:22:47.909507 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/minicluster-data/ts-0/wal/instance
I20260808 22:22:47.909775 9923 external_mini_cluster.cc:1428] Setting key dc616dd1383eba883c2f0fd21c299c13
I20260808 22:22:47.912616 25774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:47.901006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.193:45569 (local address 127.9.176.254:38921)
0808 22:22:47.901135 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:47.901139 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:47.901998 (+ 859us) server_negotiation.cc:407] Connection header received
0808 22:22:47.903320 (+ 1322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:47.903323 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:47.903372 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:47.903445 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:47.904581 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:47.905187 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:47.906343 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:47.906594 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:47.906675 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:47.909022 (+ 2347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:47.909025 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:47.909027 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:22:47.909051 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:47.910527 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:47.911014 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:47.911039 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:47.911041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:47.911140 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:47.911533 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:47.911536 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:47.911537 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:47.911722 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:47.911797 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:47.912384 (+ 587us) server_negotiation.cc:299] Negotiation successful
0808 22:22:47.912484 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 22:22:47.913673 26027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:47.901368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:38921 (local address 127.9.176.193:45569)
0808 22:22:47.901829 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:47.901873 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:47.903107 (+ 1234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:47.903620 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:47.903626 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:47.903975 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:47.904447 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:47.904460 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:47.905321 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:47.905325 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:47.906223 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:47.906233 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:47.906714 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:47.906717 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:47.906857 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:47.907363 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:47.907379 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:47.908818 (+ 1439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:22:47.910646 (+ 1828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:47.910653 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:47.910667 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:47.910916 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:47.911231 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:47.911235 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:47.911237 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:47.911430 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:47.911829 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:22:47.911835 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:22:47.912143 (+ 308us) client_negotiation.cc:769] Sending connection context
0808 22:22:47.912342 (+ 199us) client_negotiation.cc:240] Negotiation successful
0808 22:22:47.912670 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":123,"threads_started":1}
I20260808 22:22:47.914925 26024 heartbeater.cc:347] Connected to a master server at 127.9.176.254:38921
I20260808 22:22:47.915241 26024 heartbeater.cc:464] Registering TS with master...
I20260808 22:22:47.915771 26024 heartbeater.cc:511] Master 127.9.176.254:38921 requested a full tablet report, sending...
I20260808 22:22:47.917321 25714 ts_manager.cc:194] Registered new tserver with Master: 39a3df6787e74130b0aef5977074117c (127.9.176.193:43669)
I20260808 22:22:47.918427 25714 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.176.193@KRBTEST.COM'} at 127.9.176.193:45569
WARNING: no policy specified for kudu/127.9.176.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.176.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.176.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.194.keytab.
Entry for principal kudu/127.9.176.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.194.keytab.
I20260808 22:22:47.939108 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.194:0
--local_ip_for_outbound_sockets=127.9.176.194
--webserver_interface=127.9.176.194
--webserver_port=0
--tserver_master_addrs=127.9.176.254:38921
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.194.keytab
--principal=kudu/127.9.176.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:44439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:22:48.046586 26031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:22:48.046800 26031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:22:48.046838 26031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:22:48.050767 26031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:22:48.050819 26031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:22:48.050871 26031 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.194
I20260808 22:22:48.055857 26031 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:44439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.194.keytab
--principal=kudu/127.9.176.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.176.194
--webserver_port=0
--tserver_master_addrs=127.9.176.254:38921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26031
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.194
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:22:48.056747 26031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:22:48.057683 26031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:22:48.066803 26041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:48.066905 26043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:22:48.066857 26031 server_base.cc:1034] running on GCE node
W20260808 22:22:48.066799 26046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:22:48.067436 26031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:22:48.067981 26031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:22:48.069144 26031 hybrid_clock.cc:648] HybridClock initialized: now 1786227768069116 us; error 39 us; skew 500 ppm
Aug 08 22:22:48 dist-test-slave-ncp9 krb5kdc[25644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227768, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:22:48.071678 26031 init.cc:377] Logged in from keytab as kudu/127.9.176.194@KRBTEST.COM (short username kudu)
I20260808 22:22:48.072476 26031 webserver.cc:460] Webserver started at http://127.9.176.194:40841/ using document root <none> and password file <none>
I20260808 22:22:48.073015 26031 fs_manager.cc:359] Metadata directory not provided
I20260808 22:22:48.073086 26031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:22:48.073279 26031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:22:48.075150 26031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/minicluster-data/ts-1/data/instance:
uuid: "74c39b352b794a369de3658f07ef2a9e"
format_stamp: "Formatted at 2026-08-08 22:22:48 on dist-test-slave-ncp9"
server_key: "50d1d597a0aa5ecd0184a86cd1a5ef47"
server_key_iv: "959a608c506164d2d470999f8c77e5b2"
server_key_version: "encryptionkey@0"
I20260808 22:22:48.075659 26031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/minicluster-data/ts-1/wal/instance:
uuid: "74c39b352b794a369de3658f07ef2a9e"
format_stamp: "Formatted at 2026-08-08 22:22:48 on dist-test-slave-ncp9"
server_key: "50d1d597a0aa5ecd0184a86cd1a5ef47"
server_key_iv: "959a608c506164d2d470999f8c77e5b2"
server_key_version: "encryptionkey@0"
I20260808 22:22:48.079099 26031 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260808 22:22:48.081285 26063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:48.082415 26031 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 22:22:48.082530 26031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/minicluster-data/ts-1/wal
uuid: "74c39b352b794a369de3658f07ef2a9e"
format_stamp: "Formatted at 2026-08-08 22:22:48 on dist-test-slave-ncp9"
server_key: "50d1d597a0aa5ecd0184a86cd1a5ef47"
server_key_iv: "959a608c506164d2d470999f8c77e5b2"
server_key_version: "encryptionkey@0"
I20260808 22:22:48.082617 26031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:22:48.097774 26031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:22:48.100347 26031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:22:48.100512 26031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:22:48.101081 26031 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:22:48.101881 26031 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:22:48.101927 26031 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:48.101976 26031 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:22:48.102010 26031 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:48.110087 26031 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.194:44613
I20260808 22:22:48.110124 26234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.194:44613 every 8 connection(s)
I20260808 22:22:48.110910 26031 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
I20260808 22:22:48.114938 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 26031
I20260808 22:22:48.115032 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/minicluster-data/ts-1/wal/instance
I20260808 22:22:48.115294 9923 external_mini_cluster.cc:1428] Setting key 7afbffbd8a8074e72bae8246fb8fc56d
Aug 08 22:22:48 dist-test-slave-ncp9 krb5kdc[25644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227768, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.194@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
I20260808 22:22:48.124548 25774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:48.112580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.194:40289 (local address 127.9.176.254:38921)
0808 22:22:48.112696 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:48.112700 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:48.113491 (+ 791us) server_negotiation.cc:407] Connection header received
0808 22:22:48.114836 (+ 1345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:48.114840 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:48.114898 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:48.114970 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:48.116217 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:48.116951 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:48.118074 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:48.118368 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:48.118470 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:48.121010 (+ 2540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:48.121014 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:48.121017 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:22:48.121043 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:48.122635 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:48.123110 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:48.123114 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:48.123115 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:48.123213 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:48.123492 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:48.123495 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:48.123497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:48.123721 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:48.123821 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:48.124257 (+ 436us) server_negotiation.cc:299] Negotiation successful
0808 22:22:48.124385 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260808 22:22:48.125227 26239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:48.112885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:38921 (local address 127.9.176.194:40289)
0808 22:22:48.113350 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:48.113390 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:48.114630 (+ 1240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:48.115153 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:48.115159 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:48.115495 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:48.116076 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:48.116095 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:48.117071 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:48.117074 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:48.117953 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:48.117962 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:48.118509 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:48.118513 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:48.118729 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:48.119296 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:48.119316 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:48.120835 (+ 1519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:22:48.122747 (+ 1912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:48.122752 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:48.122762 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:48.123014 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:48.123293 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:48.123295 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:48.123296 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:48.123412 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:48.123810 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:22:48.123814 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:22:48.124034 (+ 220us) client_negotiation.cc:769] Sending connection context
0808 22:22:48.124175 (+ 141us) client_negotiation.cc:240] Negotiation successful
0808 22:22:48.124403 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":110,"threads_started":1}
WARNING: no policy specified for kudu/127.9.176.195@KRBTEST.COM; defaulting to no policy
I20260808 22:22:48.126248 26236 heartbeater.cc:347] Connected to a master server at 127.9.176.254:38921
I20260808 22:22:48.126443 26236 heartbeater.cc:464] Registering TS with master...
I20260808 22:22:48.126868 26236 heartbeater.cc:511] Master 127.9.176.254:38921 requested a full tablet report, sending...
I20260808 22:22:48.127825 25714 ts_manager.cc:194] Registered new tserver with Master: 74c39b352b794a369de3658f07ef2a9e (127.9.176.194:44613)
I20260808 22:22:48.128525 25714 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.176.194@KRBTEST.COM'} at 127.9.176.194:40289
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.176.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.176.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.195.keytab.
Entry for principal kudu/127.9.176.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.195.keytab.
I20260808 22:22:48.143854 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.195:0
--local_ip_for_outbound_sockets=127.9.176.195
--webserver_interface=127.9.176.195
--webserver_port=0
--tserver_master_addrs=127.9.176.254:38921
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.195.keytab
--principal=kudu/127.9.176.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:44439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:22:48.254097 26242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:22:48.254375 26242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:22:48.254419 26242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:22:48.258303 26242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:22:48.258360 26242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:22:48.258416 26242 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.195
I20260808 22:22:48.263701 26242 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:44439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.195.keytab
--principal=kudu/127.9.176.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.176.195
--webserver_port=0
--tserver_master_addrs=127.9.176.254:38921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26242
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.195
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:22:48.264691 26242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:22:48.265681 26242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:22:48.274943 26255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:48.274937 26252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:22:48.275082 26250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:22:48.275141 26242 server_base.cc:1034] running on GCE node
I20260808 22:22:48.275615 26242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:22:48.276242 26242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:22:48.277396 26242 hybrid_clock.cc:648] HybridClock initialized: now 1786227768277381 us; error 28 us; skew 500 ppm
Aug 08 22:22:48 dist-test-slave-ncp9 krb5kdc[25644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227768, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:22:48.282703 26242 init.cc:377] Logged in from keytab as kudu/127.9.176.195@KRBTEST.COM (short username kudu)
I20260808 22:22:48.283568 26242 webserver.cc:460] Webserver started at http://127.9.176.195:39149/ using document root <none> and password file <none>
I20260808 22:22:48.284102 26242 fs_manager.cc:359] Metadata directory not provided
I20260808 22:22:48.284170 26242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:22:48.284358 26242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:22:48.286410 26242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/minicluster-data/ts-2/data/instance:
uuid: "e440e974a8cb4f73a4c644b81b41bbf5"
format_stamp: "Formatted at 2026-08-08 22:22:48 on dist-test-slave-ncp9"
server_key: "19498f34c2fbc8d961ebda253b897c89"
server_key_iv: "caf04087e933b47f3f9b368506ca70fb"
server_key_version: "encryptionkey@0"
I20260808 22:22:48.286949 26242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/minicluster-data/ts-2/wal/instance:
uuid: "e440e974a8cb4f73a4c644b81b41bbf5"
format_stamp: "Formatted at 2026-08-08 22:22:48 on dist-test-slave-ncp9"
server_key: "19498f34c2fbc8d961ebda253b897c89"
server_key_iv: "caf04087e933b47f3f9b368506ca70fb"
server_key_version: "encryptionkey@0"
I20260808 22:22:48.290737 26242 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 22:22:48.293104 26267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:48.294262 26242 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 22:22:48.294371 26242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/minicluster-data/ts-2/wal
uuid: "e440e974a8cb4f73a4c644b81b41bbf5"
format_stamp: "Formatted at 2026-08-08 22:22:48 on dist-test-slave-ncp9"
server_key: "19498f34c2fbc8d961ebda253b897c89"
server_key_iv: "caf04087e933b47f3f9b368506ca70fb"
server_key_version: "encryptionkey@0"
I20260808 22:22:48.294461 26242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:22:48.306133 26242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:22:48.308743 26242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:22:48.308933 26242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:22:48.309487 26242 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:22:48.310293 26242 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:22:48.310341 26242 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:48.310392 26242 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:22:48.310420 26242 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:22:48.320752 26242 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.195:45463
I20260808 22:22:48.320782 26447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.195:45463 every 8 connection(s)
I20260808 22:22:48.321676 26242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
I20260808 22:22:48.330219 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 26242
I20260808 22:22:48.330327 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/minicluster-data/ts-2/wal/instance
I20260808 22:22:48.330621 9923 external_mini_cluster.cc:1428] Setting key 3363a51ee8d1e2f34bc1f00f11a356a3
Aug 08 22:22:48 dist-test-slave-ncp9 krb5kdc[25644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227768, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.195@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
I20260808 22:22:48.334945 25774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:48.323474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.195:60859 (local address 127.9.176.254:38921)
0808 22:22:48.323603 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:48.323607 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:48.324462 (+ 855us) server_negotiation.cc:407] Connection header received
0808 22:22:48.325859 (+ 1397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:48.325862 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:48.325918 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:48.325993 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:48.327148 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:48.327688 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:48.328813 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:48.329103 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:48.329193 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:48.331634 (+ 2441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:48.331637 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:48.331639 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:22:48.331664 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:48.333074 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:48.333524 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:48.333528 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:48.333529 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:48.333622 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:48.333933 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:48.333936 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:48.333938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:48.334115 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:48.334199 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:48.334671 (+ 472us) server_negotiation.cc:299] Negotiation successful
0808 22:22:48.334791 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260808 22:22:48.335685 26452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:48.323876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:38921 (local address 127.9.176.195:60859)
0808 22:22:48.324310 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:48.324352 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:48.325645 (+ 1293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:48.326178 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:48.326185 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:48.326533 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:22:48.326996 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:48.327010 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:48.327806 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:48.327809 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:48.328695 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:48.328704 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:48.329254 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:48.329258 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:48.329442 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:48.329960 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:22:48.329975 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:22:48.331483 (+ 1508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:22:48.333174 (+ 1691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:48.333180 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:48.333194 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:48.333432 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:48.333704 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:22:48.333707 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:22:48.333709 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:22:48.333845 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:22:48.334214 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:22:48.334221 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:22:48.334450 (+ 229us) client_negotiation.cc:769] Sending connection context
0808 22:22:48.334612 (+ 162us) client_negotiation.cc:240] Negotiation successful
0808 22:22:48.334917 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":120,"threads_started":1}
I20260808 22:22:48.336681 26450 heartbeater.cc:347] Connected to a master server at 127.9.176.254:38921
I20260808 22:22:48.336997 26450 heartbeater.cc:464] Registering TS with master...
I20260808 22:22:48.337484 26450 heartbeater.cc:511] Master 127.9.176.254:38921 requested a full tablet report, sending...
I20260808 22:22:48.338464 25714 ts_manager.cc:194] Registered new tserver with Master: e440e974a8cb4f73a4c644b81b41bbf5 (127.9.176.195:45463)
I20260808 22:22:48.339008 25714 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.176.195@KRBTEST.COM'} at 127.9.176.195:60859
I20260808 22:22:48.344225 9923 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 22:22:48.352840 25774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:48.346051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50146 (local address 127.9.176.254:38921)
0808 22:22:48.346174 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:48.346178 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:48.346350 (+ 172us) server_negotiation.cc:407] Connection header received
0808 22:22:48.346470 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:48.346472 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:48.346516 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:48.346578 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:22:48.347251 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:48.347737 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:48.348600 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:48.348805 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:48.348914 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:48.350163 (+ 1249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:22:48.350166 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:22:48.350168 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:22:48.350189 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:22:48.351309 (+ 1120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:48.351704 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:48.351708 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:48.351710 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:48.351781 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:22:48.352026 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:22:48.352030 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:22:48.352031 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:22:48.352234 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:22:48.352351 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:48.352596 (+ 245us) server_negotiation.cc:299] Negotiation successful
0808 22:22:48.352698 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 22:22:48.355525 25714 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50146:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 22:22:48.358446 25714 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 22:22:48.372495 26467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:48.366447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.194:44613 (local address 127.0.0.1:53226)
0808 22:22:48.367196 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:48.367247 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:48.367369 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:48.368651 (+ 1282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:48.368654 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:48.368671 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:22:48.368751 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:48.368756 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:48.370090 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:48.370093 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:48.371010 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:48.371020 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:48.372020 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:48.372023 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:48.372107 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:48.372137 (+ 30us) client_negotiation.cc:769] Sending connection context
0808 22:22:48.372236 (+ 99us) client_negotiation.cc:240] Negotiation successful
0808 22:22:48.372348 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":667,"spinlock_wait_cycles":1280,"thread_start_us":98,"threads_started":1}
I20260808 22:22:48.372501 26470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:48.366813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53226 (local address 127.9.176.194:44613)
0808 22:22:48.367242 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:48.367254 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:48.367317 (+ 63us) server_negotiation.cc:407] Connection header received
0808 22:22:48.368241 (+ 924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:48.368246 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:48.368419 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:48.368557 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:22:48.368856 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:48.369928 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:48.371140 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:48.371914 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:48.372027 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:48.372176 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:48.372263 (+ 87us) server_negotiation.cc:299] Negotiation successful
0808 22:22:48.372357 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":105,"threads_started":1}
I20260808 22:22:48.372798 26474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:48.367663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53576 (local address 127.9.176.195:45463)
0808 22:22:48.368056 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:48.368067 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:48.368088 (+ 21us) server_negotiation.cc:407] Connection header received
0808 22:22:48.368155 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:48.368160 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:48.368351 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:48.368491 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:22:48.368695 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:48.369502 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:48.371337 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:48.372109 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:48.372206 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:48.372335 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:48.372571 (+ 236us) server_negotiation.cc:299] Negotiation successful
0808 22:22:48.372675 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":119,"threads_started":1}
I20260808 22:22:48.373041 26472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:48.367112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.195:45463 (local address 127.0.0.1:53576)
0808 22:22:48.367767 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:48.367781 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:48.367864 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:48.368472 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:48.368475 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:48.368491 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:22:48.368575 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:48.368582 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:48.369635 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:48.369638 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:48.371209 (+ 1571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:48.371221 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:48.372218 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:48.372222 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:48.372311 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:48.372331 (+ 20us) client_negotiation.cc:769] Sending connection context
0808 22:22:48.372393 (+ 62us) client_negotiation.cc:240] Negotiation successful
0808 22:22:48.372450 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"mutex_wait_us":133,"thread_start_us":84,"threads_started":1}
I20260808 22:22:48.373724 26468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:48.366384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.193:43669 (local address 127.0.0.1:46156)
0808 22:22:48.367196 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:48.367247 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:48.367369 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:48.368096 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:48.368103 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:48.368127 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:22:48.368256 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:48.368262 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:48.369668 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:48.369670 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:48.371211 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:48.371221 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:48.373333 (+ 2112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:48.373337 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:48.373426 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:48.373450 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 22:22:48.373530 (+ 80us) client_negotiation.cc:240] Negotiation successful
0808 22:22:48.373602 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"mutex_wait_us":105,"thread_start_us":72,"threads_started":1}
I20260808 22:22:48.374298 26469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:48.366522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46156 (local address 127.9.176.193:43669)
0808 22:22:48.367626 (+ 1104us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:48.367645 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:48.367667 (+ 22us) server_negotiation.cc:407] Connection header received
0808 22:22:48.367746 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:48.367751 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:48.367945 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:48.368071 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:22:48.368374 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:48.369551 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:48.372395 (+ 2844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:48.373198 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:48.373330 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:48.373462 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:48.374066 (+ 604us) server_negotiation.cc:299] Negotiation successful
0808 22:22:48.374165 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":998,"thread_start_us":81,"threads_started":1}
I20260808 22:22:48.375996 26348 tablet_service.cc:1467] Processing CreateTablet for tablet c44b75f8e09d4c87a0212dbcea932bc5 (DEFAULT_TABLE table=test-table [id=af91479172e442778729a95f18501579]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:22:48.376103 26140 tablet_service.cc:1467] Processing CreateTablet for tablet c44b75f8e09d4c87a0212dbcea932bc5 (DEFAULT_TABLE table=test-table [id=af91479172e442778729a95f18501579]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:22:48.377024 26348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c44b75f8e09d4c87a0212dbcea932bc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:22:48.377156 26140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c44b75f8e09d4c87a0212dbcea932bc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:22:48.376808 25920 tablet_service.cc:1467] Processing CreateTablet for tablet c44b75f8e09d4c87a0212dbcea932bc5 (DEFAULT_TABLE table=test-table [id=af91479172e442778729a95f18501579]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:22:48.377748 25920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c44b75f8e09d4c87a0212dbcea932bc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:22:48.382908 26475 tablet_bootstrap.cc:492] T c44b75f8e09d4c87a0212dbcea932bc5 P e440e974a8cb4f73a4c644b81b41bbf5: Bootstrap starting.
I20260808 22:22:48.384379 26478 tablet_bootstrap.cc:492] T c44b75f8e09d4c87a0212dbcea932bc5 P 74c39b352b794a369de3658f07ef2a9e: Bootstrap starting.
I20260808 22:22:48.385241 26479 tablet_bootstrap.cc:492] T c44b75f8e09d4c87a0212dbcea932bc5 P 39a3df6787e74130b0aef5977074117c: Bootstrap starting.
I20260808 22:22:48.385692 26475 tablet_bootstrap.cc:654] T c44b75f8e09d4c87a0212dbcea932bc5 P e440e974a8cb4f73a4c644b81b41bbf5: Neither blocks nor log segments found. Creating new log.
I20260808 22:22:48.386555 26475 log.cc:824] T c44b75f8e09d4c87a0212dbcea932bc5 P e440e974a8cb4f73a4c644b81b41bbf5: Log is configured to *not* fsync() on all Append() calls
I20260808 22:22:48.386814 26478 tablet_bootstrap.cc:654] T c44b75f8e09d4c87a0212dbcea932bc5 P 74c39b352b794a369de3658f07ef2a9e: Neither blocks nor log segments found. Creating new log.
I20260808 22:22:48.387238 26479 tablet_bootstrap.cc:654] T c44b75f8e09d4c87a0212dbcea932bc5 P 39a3df6787e74130b0aef5977074117c: Neither blocks nor log segments found. Creating new log.
I20260808 22:22:48.387639 26478 log.cc:824] T c44b75f8e09d4c87a0212dbcea932bc5 P 74c39b352b794a369de3658f07ef2a9e: Log is configured to *not* fsync() on all Append() calls
I20260808 22:22:48.388331 26479 log.cc:824] T c44b75f8e09d4c87a0212dbcea932bc5 P 39a3df6787e74130b0aef5977074117c: Log is configured to *not* fsync() on all Append() calls
I20260808 22:22:48.388651 26475 tablet_bootstrap.cc:492] T c44b75f8e09d4c87a0212dbcea932bc5 P e440e974a8cb4f73a4c644b81b41bbf5: No bootstrap required, opened a new log
I20260808 22:22:48.388847 26475 ts_tablet_manager.cc:1397] T c44b75f8e09d4c87a0212dbcea932bc5 P e440e974a8cb4f73a4c644b81b41bbf5: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 22:22:48.389539 26478 tablet_bootstrap.cc:492] T c44b75f8e09d4c87a0212dbcea932bc5 P 74c39b352b794a369de3658f07ef2a9e: No bootstrap required, opened a new log
I20260808 22:22:48.389732 26478 ts_tablet_manager.cc:1397] T c44b75f8e09d4c87a0212dbcea932bc5 P 74c39b352b794a369de3658f07ef2a9e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 22:22:48.390316 26479 tablet_bootstrap.cc:492] T c44b75f8e09d4c87a0212dbcea932bc5 P 39a3df6787e74130b0aef5977074117c: No bootstrap required, opened a new log
I20260808 22:22:48.390494 26479 ts_tablet_manager.cc:1397] T c44b75f8e09d4c87a0212dbcea932bc5 P 39a3df6787e74130b0aef5977074117c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260808 22:22:48.392004 26475 raft_consensus.cc:348] T c44b75f8e09d4c87a0212dbcea932bc5 P e440e974a8cb4f73a4c644b81b41bbf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39a3df6787e74130b0aef5977074117c" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 43669 } } peers { permanent_uuid: "74c39b352b794a369de3658f07ef2a9e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 44613 } } peers { permanent_uuid: "e440e974a8cb4f73a4c644b81b41bbf5" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45463 } }
I20260808 22:22:48.392220 26475 raft_consensus.cc:374] T c44b75f8e09d4c87a0212dbcea932bc5 P e440e974a8cb4f73a4c644b81b41bbf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:22:48.392269 26475 raft_consensus.cc:729] T c44b75f8e09d4c87a0212dbcea932bc5 P e440e974a8cb4f73a4c644b81b41bbf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e440e974a8cb4f73a4c644b81b41bbf5, State: Initialized, Role: FOLLOWER
I20260808 22:22:48.392660 26475 consensus_queue.cc:260] T c44b75f8e09d4c87a0212dbcea932bc5 P e440e974a8cb4f73a4c644b81b41bbf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39a3df6787e74130b0aef5977074117c" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 43669 } } peers { permanent_uuid: "74c39b352b794a369de3658f07ef2a9e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 44613 } } peers { permanent_uuid: "e440e974a8cb4f73a4c644b81b41bbf5" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45463 } }
I20260808 22:22:48.393268 26450 heartbeater.cc:503] Master 127.9.176.254:38921 was elected leader, sending a full tablet report...
I20260808 22:22:48.393380 26478 raft_consensus.cc:348] T c44b75f8e09d4c87a0212dbcea932bc5 P 74c39b352b794a369de3658f07ef2a9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39a3df6787e74130b0aef5977074117c" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 43669 } } peers { permanent_uuid: "74c39b352b794a369de3658f07ef2a9e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 44613 } } peers { permanent_uuid: "e440e974a8cb4f73a4c644b81b41bbf5" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45463 } }
I20260808 22:22:48.393617 26475 ts_tablet_manager.cc:1428] T c44b75f8e09d4c87a0212dbcea932bc5 P e440e974a8cb4f73a4c644b81b41bbf5: Time spent starting tablet: real 0.005s user 0.000s sys 0.004s
I20260808 22:22:48.393626 26478 raft_consensus.cc:374] T c44b75f8e09d4c87a0212dbcea932bc5 P 74c39b352b794a369de3658f07ef2a9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:22:48.393674 26478 raft_consensus.cc:729] T c44b75f8e09d4c87a0212dbcea932bc5 P 74c39b352b794a369de3658f07ef2a9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74c39b352b794a369de3658f07ef2a9e, State: Initialized, Role: FOLLOWER
I20260808 22:22:48.394096 26478 consensus_queue.cc:260] T c44b75f8e09d4c87a0212dbcea932bc5 P 74c39b352b794a369de3658f07ef2a9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39a3df6787e74130b0aef5977074117c" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 43669 } } peers { permanent_uuid: "74c39b352b794a369de3658f07ef2a9e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 44613 } } peers { permanent_uuid: "e440e974a8cb4f73a4c644b81b41bbf5" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45463 } }
I20260808 22:22:48.394086 26479 raft_consensus.cc:348] T c44b75f8e09d4c87a0212dbcea932bc5 P 39a3df6787e74130b0aef5977074117c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39a3df6787e74130b0aef5977074117c" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 43669 } } peers { permanent_uuid: "74c39b352b794a369de3658f07ef2a9e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 44613 } } peers { permanent_uuid: "e440e974a8cb4f73a4c644b81b41bbf5" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45463 } }
I20260808 22:22:48.394309 26479 raft_consensus.cc:374] T c44b75f8e09d4c87a0212dbcea932bc5 P 39a3df6787e74130b0aef5977074117c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:22:48.394358 26479 raft_consensus.cc:729] T c44b75f8e09d4c87a0212dbcea932bc5 P 39a3df6787e74130b0aef5977074117c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39a3df6787e74130b0aef5977074117c, State: Initialized, Role: FOLLOWER
I20260808 22:22:48.394769 26479 consensus_queue.cc:260] T c44b75f8e09d4c87a0212dbcea932bc5 P 39a3df6787e74130b0aef5977074117c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39a3df6787e74130b0aef5977074117c" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 43669 } } peers { permanent_uuid: "74c39b352b794a369de3658f07ef2a9e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 44613 } } peers { permanent_uuid: "e440e974a8cb4f73a4c644b81b41bbf5" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45463 } }
I20260808 22:22:48.395009 26478 ts_tablet_manager.cc:1428] T c44b75f8e09d4c87a0212dbcea932bc5 P 74c39b352b794a369de3658f07ef2a9e: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260808 22:22:48.395296 26236 heartbeater.cc:503] Master 127.9.176.254:38921 was elected leader, sending a full tablet report...
I20260808 22:22:48.395515 26024 heartbeater.cc:503] Master 127.9.176.254:38921 was elected leader, sending a full tablet report...
I20260808 22:22:48.395637 26479 ts_tablet_manager.cc:1428] T c44b75f8e09d4c87a0212dbcea932bc5 P 39a3df6787e74130b0aef5977074117c: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260808 22:22:48.400210 26026 tablet.cc:2369] T c44b75f8e09d4c87a0212dbcea932bc5 P 39a3df6787e74130b0aef5977074117c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 22:22:48.533233 26488 raft_consensus.cc:482] T c44b75f8e09d4c87a0212dbcea932bc5 P 39a3df6787e74130b0aef5977074117c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 22:22:48.533452 26488 raft_consensus.cc:504] T c44b75f8e09d4c87a0212dbcea932bc5 P 39a3df6787e74130b0aef5977074117c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39a3df6787e74130b0aef5977074117c" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 43669 } } peers { permanent_uuid: "74c39b352b794a369de3658f07ef2a9e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 44613 } } peers { permanent_uuid: "e440e974a8cb4f73a4c644b81b41bbf5" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45463 } }
I20260808 22:22:48.534580 26488 leader_election.cc:288] T c44b75f8e09d4c87a0212dbcea932bc5 P 39a3df6787e74130b0aef5977074117c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74c39b352b794a369de3658f07ef2a9e (127.9.176.194:44613), e440e974a8cb4f73a4c644b81b41bbf5 (127.9.176.195:45463)
I20260808 22:22:48.538501 26470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:48.534895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.193:38299 (local address 127.9.176.194:44613)
0808 22:22:48.535046 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:48.535051 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:48.535145 (+ 94us) server_negotiation.cc:407] Connection header received
0808 22:22:48.535295 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:48.535298 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:48.535357 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:48.535438 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:22:48.535721 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:48.536279 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:48.537526 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:48.537983 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:48.538094 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:48.538150 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:48.538320 (+ 170us) server_negotiation.cc:299] Negotiation successful
0808 22:22:48.538390 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 22:22:48.538515 26489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:48.534799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.194:44613 (local address 127.9.176.193:38299)
0808 22:22:48.535059 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:48.535073 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:48.535184 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:48.535480 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:48.535483 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:48.535503 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:22:48.535619 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:48.535624 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:48.536380 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:48.536384 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:48.537412 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:48.537421 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:48.538099 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:48.538103 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:48.538198 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:48.538222 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 22:22:48.538291 (+ 69us) client_negotiation.cc:240] Negotiation successful
0808 22:22:48.538361 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":96,"threads_started":1}
I20260808 22:22:48.538681 26474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:48.535340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.193:35689 (local address 127.9.176.195:45463)
0808 22:22:48.535459 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:48.535464 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:48.535565 (+ 101us) server_negotiation.cc:407] Connection header received
0808 22:22:48.535681 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:48.535684 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:48.535729 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:48.535790 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:22:48.536041 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:48.536643 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:48.537781 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:48.538183 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:48.538290 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:48.538342 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:48.538510 (+ 168us) server_negotiation.cc:299] Negotiation successful
0808 22:22:48.538575 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 22:22:48.538695 26490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:48.535213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.195:45463 (local address 127.9.176.193:35689)
0808 22:22:48.535477 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:48.535490 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 22:22:48.535589 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:22:48.535809 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:22:48.535811 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:22:48.535827 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:22:48.535935 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:48.535940 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:48.536748 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:48.536751 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:48.537615 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:22:48.537625 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:22:48.538294 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:22:48.538298 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:22:48.538401 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:48.538420 (+ 19us) client_negotiation.cc:769] Sending connection context
0808 22:22:48.538483 (+ 63us) client_negotiation.cc:240] Negotiation successful
0808 22:22:48.538558 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":112,"threads_started":1}
I20260808 22:22:48.539125 26175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c44b75f8e09d4c87a0212dbcea932bc5" candidate_uuid: "39a3df6787e74130b0aef5977074117c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74c39b352b794a369de3658f07ef2a9e" is_pre_election: true
I20260808 22:22:48.539206 26380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c44b75f8e09d4c87a0212dbcea932bc5" candidate_uuid: "39a3df6787e74130b0aef5977074117c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e440e974a8cb4f73a4c644b81b41bbf5" is_pre_election: true
I20260808 22:22:48.539475 26175 raft_consensus.cc:2454] T c44b75f8e09d4c87a0212dbcea932bc5 P 74c39b352b794a369de3658f07ef2a9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39a3df6787e74130b0aef5977074117c in term 0.
I20260808 22:22:48.539475 26380 raft_consensus.cc:2454] T c44b75f8e09d4c87a0212dbcea932bc5 P e440e974a8cb4f73a4c644b81b41bbf5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39a3df6787e74130b0aef5977074117c in term 0.
I20260808 22:22:48.539932 25848 leader_election.cc:302] T c44b75f8e09d4c87a0212dbcea932bc5 P 39a3df6787e74130b0aef5977074117c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39a3df6787e74130b0aef5977074117c, e440e974a8cb4f73a4c644b81b41bbf5; no voters:
I20260808 22:22:48.540169 26488 raft_consensus.cc:2781] T c44b75f8e09d4c87a0212dbcea932bc5 P 39a3df6787e74130b0aef5977074117c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 22:22:48.540251 26488 raft_consensus.cc:482] T c44b75f8e09d4c87a0212dbcea932bc5 P 39a3df6787e74130b0aef5977074117c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 22:22:48.540299 26488 raft_consensus.cc:3037] T c44b75f8e09d4c87a0212dbcea932bc5 P 39a3df6787e74130b0aef5977074117c [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:22:48.541378 26488 raft_consensus.cc:504] T c44b75f8e09d4c87a0212dbcea932bc5 P 39a3df6787e74130b0aef5977074117c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39a3df6787e74130b0aef5977074117c" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 43669 } } peers { permanent_uuid: "74c39b352b794a369de3658f07ef2a9e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 44613 } } peers { permanent_uuid: "e440e974a8cb4f73a4c644b81b41bbf5" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45463 } }
I20260808 22:22:48.541692 26488 leader_election.cc:288] T c44b75f8e09d4c87a0212dbcea932bc5 P 39a3df6787e74130b0aef5977074117c [CANDIDATE]: Term 1 election: Requested vote from peers 74c39b352b794a369de3658f07ef2a9e (127.9.176.194:44613), e440e974a8cb4f73a4c644b81b41bbf5 (127.9.176.195:45463)
I20260808 22:22:48.542043 26175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c44b75f8e09d4c87a0212dbcea932bc5" candidate_uuid: "39a3df6787e74130b0aef5977074117c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74c39b352b794a369de3658f07ef2a9e"
I20260808 22:22:48.542096 26380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c44b75f8e09d4c87a0212dbcea932bc5" candidate_uuid: "39a3df6787e74130b0aef5977074117c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e440e974a8cb4f73a4c644b81b41bbf5"
I20260808 22:22:48.542166 26175 raft_consensus.cc:3037] T c44b75f8e09d4c87a0212dbcea932bc5 P 74c39b352b794a369de3658f07ef2a9e [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:22:48.542227 26380 raft_consensus.cc:3037] T c44b75f8e09d4c87a0212dbcea932bc5 P e440e974a8cb4f73a4c644b81b41bbf5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:22:48.543156 26175 raft_consensus.cc:2454] T c44b75f8e09d4c87a0212dbcea932bc5 P 74c39b352b794a369de3658f07ef2a9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39a3df6787e74130b0aef5977074117c in term 1.
I20260808 22:22:48.543211 26380 raft_consensus.cc:2454] T c44b75f8e09d4c87a0212dbcea932bc5 P e440e974a8cb4f73a4c644b81b41bbf5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39a3df6787e74130b0aef5977074117c in term 1.
I20260808 22:22:48.543411 25848 leader_election.cc:302] T c44b75f8e09d4c87a0212dbcea932bc5 P 39a3df6787e74130b0aef5977074117c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39a3df6787e74130b0aef5977074117c, 74c39b352b794a369de3658f07ef2a9e; no voters:
I20260808 22:22:48.543574 26488 raft_consensus.cc:2781] T c44b75f8e09d4c87a0212dbcea932bc5 P 39a3df6787e74130b0aef5977074117c [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:22:48.543769 26488 raft_consensus.cc:686] T c44b75f8e09d4c87a0212dbcea932bc5 P 39a3df6787e74130b0aef5977074117c [term 1 LEADER]: Becoming Leader. State: Replica: 39a3df6787e74130b0aef5977074117c, State: Running, Role: LEADER
I20260808 22:22:48.544121 26488 consensus_queue.cc:237] T c44b75f8e09d4c87a0212dbcea932bc5 P 39a3df6787e74130b0aef5977074117c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39a3df6787e74130b0aef5977074117c" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 43669 } } peers { permanent_uuid: "74c39b352b794a369de3658f07ef2a9e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 44613 } } peers { permanent_uuid: "e440e974a8cb4f73a4c644b81b41bbf5" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45463 } }
I20260808 22:22:48.547191 25713 catalog_manager.cc:5495] T c44b75f8e09d4c87a0212dbcea932bc5 P 39a3df6787e74130b0aef5977074117c reported cstate change: term changed from 0 to 1, leader changed from <none> to 39a3df6787e74130b0aef5977074117c (127.9.176.193). New cstate: current_term: 1 leader_uuid: "39a3df6787e74130b0aef5977074117c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39a3df6787e74130b0aef5977074117c" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 43669 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74c39b352b794a369de3658f07ef2a9e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 44613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e440e974a8cb4f73a4c644b81b41bbf5" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45463 } health_report { overall_health: UNKNOWN } } }
I20260808 22:22:48.558322 26469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:48.555017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46170 (local address 127.9.176.193:43669)
0808 22:22:48.555144 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0808 22:22:48.555149 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:22:48.555161 (+ 12us) server_negotiation.cc:407] Connection header received
0808 22:22:48.555264 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:22:48.555267 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:22:48.555317 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:22:48.555405 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:22:48.555644 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:48.556133 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:48.556990 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:22:48.557210 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:22:48.557294 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:22:48.557575 (+ 281us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:22:48.557963 (+ 388us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:22:48.558076 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0808 22:22:48.558144 (+ 68us) server_negotiation.cc:299] Negotiation successful
0808 22:22:48.558197 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 22:22:48.563768 26380 raft_consensus.cc:1261] T c44b75f8e09d4c87a0212dbcea932bc5 P e440e974a8cb4f73a4c644b81b41bbf5 [term 1 FOLLOWER]: Refusing update from remote peer 39a3df6787e74130b0aef5977074117c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:22:48.563768 26175 raft_consensus.cc:1261] T c44b75f8e09d4c87a0212dbcea932bc5 P 74c39b352b794a369de3658f07ef2a9e [term 1 FOLLOWER]: Refusing update from remote peer 39a3df6787e74130b0aef5977074117c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:22:48.564458 26493 consensus_queue.cc:1035] T c44b75f8e09d4c87a0212dbcea932bc5 P 39a3df6787e74130b0aef5977074117c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e440e974a8cb4f73a4c644b81b41bbf5" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:22:48.564637 26488 consensus_queue.cc:1035] T c44b75f8e09d4c87a0212dbcea932bc5 P 39a3df6787e74130b0aef5977074117c [LEADER]: Connected to new peer: Peer: permanent_uuid: "74c39b352b794a369de3658f07ef2a9e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 44613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:22:48.571991 26501 mvcc.cc:205] Tried to move back new op lower bound from 7316388940030558208 to 7316388939959218176. Current Snapshot: MvccSnapshot[applied={T|T < 7316388940030558208}]
I20260808 22:22:48.572202 26500 mvcc.cc:205] Tried to move back new op lower bound from 7316388940030558208 to 7316388939959218176. Current Snapshot: MvccSnapshot[applied={T|T < 7316388940030558208}]
I20260808 22:22:48.579919 25714 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50146:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:22:48.580107 25714 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50146:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:22:48.582453 25714 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9a16ed4c823642c79c71704dd9471cdb: Sending DeleteTablet for 3 replicas of tablet c44b75f8e09d4c87a0212dbcea932bc5
I20260808 22:22:48.583204 25920 tablet_service.cc:1514] Processing DeleteTablet for tablet c44b75f8e09d4c87a0212dbcea932bc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:22:48 UTC) from {username='kudu', principal='kudu/127.9.176.254@KRBTEST.COM'} at 127.0.0.1:46156
I20260808 22:22:48.583236 26348 tablet_service.cc:1514] Processing DeleteTablet for tablet c44b75f8e09d4c87a0212dbcea932bc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:22:48 UTC) from {username='kudu', principal='kudu/127.9.176.254@KRBTEST.COM'} at 127.0.0.1:53576
I20260808 22:22:48.583364 26140 tablet_service.cc:1514] Processing DeleteTablet for tablet c44b75f8e09d4c87a0212dbcea932bc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:22:48 UTC) from {username='kudu', principal='kudu/127.9.176.254@KRBTEST.COM'} at 127.0.0.1:53226
I20260808 22:22:48.583667 26512 tablet_replica.cc:330] T c44b75f8e09d4c87a0212dbcea932bc5 P 39a3df6787e74130b0aef5977074117c: stopping tablet replica
I20260808 22:22:48.583976 26512 raft_consensus.cc:2229] T c44b75f8e09d4c87a0212dbcea932bc5 P 39a3df6787e74130b0aef5977074117c [term 1 LEADER]: Raft consensus shutting down.
I20260808 22:22:48.584197 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 25817
I20260808 22:22:48.584344 26512 raft_consensus.cc:2258] T c44b75f8e09d4c87a0212dbcea932bc5 P 39a3df6787e74130b0aef5977074117c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:22:48.584851 26518 tablet_replica.cc:330] T c44b75f8e09d4c87a0212dbcea932bc5 P 74c39b352b794a369de3658f07ef2a9e: stopping tablet replica
I20260808 22:22:48.585146 26518 raft_consensus.cc:2229] T c44b75f8e09d4c87a0212dbcea932bc5 P 74c39b352b794a369de3658f07ef2a9e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:22:48.585393 26518 raft_consensus.cc:2258] T c44b75f8e09d4c87a0212dbcea932bc5 P 74c39b352b794a369de3658f07ef2a9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:22:48.589375 26518 ts_tablet_manager.cc:1905] T c44b75f8e09d4c87a0212dbcea932bc5 P 74c39b352b794a369de3658f07ef2a9e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:22:48.590432 26515 tablet_replica.cc:330] T c44b75f8e09d4c87a0212dbcea932bc5 P e440e974a8cb4f73a4c644b81b41bbf5: stopping tablet replica
I20260808 22:22:48.590770 26515 raft_consensus.cc:2229] T c44b75f8e09d4c87a0212dbcea932bc5 P e440e974a8cb4f73a4c644b81b41bbf5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:22:48.591105 26515 raft_consensus.cc:2258] T c44b75f8e09d4c87a0212dbcea932bc5 P e440e974a8cb4f73a4c644b81b41bbf5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:22:48.592348 26518 ts_tablet_manager.cc:1918] T c44b75f8e09d4c87a0212dbcea932bc5 P 74c39b352b794a369de3658f07ef2a9e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 22:22:48.592451 26518 log.cc:1196] T c44b75f8e09d4c87a0212dbcea932bc5 P 74c39b352b794a369de3658f07ef2a9e: Deleting WAL directory at /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/minicluster-data/ts-1/wal/wals/c44b75f8e09d4c87a0212dbcea932bc5
I20260808 22:22:48.592650 26515 ts_tablet_manager.cc:1905] T c44b75f8e09d4c87a0212dbcea932bc5 P e440e974a8cb4f73a4c644b81b41bbf5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:22:48.592758 26518 ts_tablet_manager.cc:1939] T c44b75f8e09d4c87a0212dbcea932bc5 P 74c39b352b794a369de3658f07ef2a9e: Deleting consensus metadata
I20260808 22:22:48.594089 25693 catalog_manager.cc:4841] TS 74c39b352b794a369de3658f07ef2a9e (127.9.176.194:44613): tablet c44b75f8e09d4c87a0212dbcea932bc5 (table test-table [id=af91479172e442778729a95f18501579]) successfully deleted
I20260808 22:22:48.595072 26515 ts_tablet_manager.cc:1918] T c44b75f8e09d4c87a0212dbcea932bc5 P e440e974a8cb4f73a4c644b81b41bbf5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 22:22:48.595152 26515 log.cc:1196] T c44b75f8e09d4c87a0212dbcea932bc5 P e440e974a8cb4f73a4c644b81b41bbf5: Deleting WAL directory at /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786227706282457-9923-0/minicluster-data/ts-2/wal/wals/c44b75f8e09d4c87a0212dbcea932bc5
I20260808 22:22:48.595405 26515 ts_tablet_manager.cc:1939] T c44b75f8e09d4c87a0212dbcea932bc5 P e440e974a8cb4f73a4c644b81b41bbf5: Deleting consensus metadata
W20260808 22:22:48.595994 25691 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.176.193:43669 (error 108)
I20260808 22:22:48.596515 25693 catalog_manager.cc:4841] TS e440e974a8cb4f73a4c644b81b41bbf5 (127.9.176.195:45463): tablet c44b75f8e09d4c87a0212dbcea932bc5 (table test-table [id=af91479172e442778729a95f18501579]) successfully deleted
I20260808 22:22:48.596829 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 26031
I20260808 22:22:48.597013 26467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:22:48.596716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.193:43669 (local address 127.0.0.1:46174)
0808 22:22:48.596841 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0808 22:22:48.596933 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.193:43669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260808 22:22:48.605523 25691 catalog_manager.cc:4584] TS 39a3df6787e74130b0aef5977074117c (127.9.176.193:43669): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c44b75f8e09d4c87a0212dbcea932bc5: Network error: Client connection negotiation failed: client connection to 127.9.176.193:43669: connect: Connection refused (error 111)
I20260808 22:22:48.610244 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 26242
I20260808 22:22:48.622628 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 25663
2026-08-08T22:22:48Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2435 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 22:22:48 dist-test-slave-ncp9 krb5kdc[26523](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 22:22:48 dist-test-slave-ncp9 krb5kdc[26523](info): set up 2 sockets
Aug 08 22:22:48 dist-test-slave-ncp9 krb5kdc[26523](info): commencing operation
krb5kdc: starting...
W20260808 22:22:49.747524 9923 mini_kdc.cc:121] Time spent starting KDC: real 1.098s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/krb5kdc/test-user.keytab.
Aug 08 22:22:49 dist-test-slave-ncp9 krb5kdc[26523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227769, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T22:22:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:22:49Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.9.176.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.9.176.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.9.176.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/krb5kdc/rangeradmin_127.9.176.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.9.176.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/krb5kdc/rangeradmin_127.9.176.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.9.176.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.9.176.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.9.176.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/krb5kdc/rangerlookup_127.9.176.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.9.176.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/krb5kdc/rangerlookup_127.9.176.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.9.176.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.176.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.176.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/krb5kdc/HTTP_127.9.176.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.9.176.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/krb5kdc/HTTP_127.9.176.212@KRBTEST.COM.keytab.
I20260808 22:22:49.903787 9923 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskqqDyFd/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/minicluster-data/postgres -l logfile start
2026-08-08 22:22:52.214 UTC [26567] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-08 22:22:52.214 UTC [26567] LOG: listening on IPv4 address "127.9.176.212", port 59827
2026-08-08 22:22:52.218 UTC [26567] LOG: listening on Unix socket "/tmp/.s.PGSQL.59827"
2026-08-08 22:22:52.244 UTC [26573] LOG: database system was shut down at 2026-08-08 22:22:50 UTC
2026-08-08 22:22:52.248 UTC [26567] LOG: database system is ready to accept connections
I20260808 22:22:53.273957 9923 mini_postgres.cc:96] Postgres bound to 59827
2026-08-08 22:22:53.281 UTC [26583] FATAL: database "slave" does not exist
127.9.176.212:59827 - accepting connections
I20260808 22:22:53.281780 9923 mini_ranger.cc:161] Starting Ranger...
I20260808 22:22:53.296582 9923 mini_ranger.cc:84] Created miniranger Postgres user
I20260808 22:22:53.707835 9923 mini_ranger.cc:87] Created ranger Postgres database
I20260808 22:22:53.707934 9923 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/minicluster-data/ranger-admin
2026-08-08 22:22:54,003 [I] DB FLAVOR :POSTGRES
2026-08-08 22:22:54,004 [I] --------- Verifying Ranger DB connection ---------
2026-08-08 22:22:54,004 [I] Checking connection..
2026-08-08 22:22:54,004 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqqDyFd/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqqDyFd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.176.212:59827/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-08 22:22:54,259 [I] Checking connection passed.
2026-08-08 22:22:54,259 [I] --------- Verifying version history table ---------
2026-08-08 22:22:54,260 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqqDyFd/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqqDyFd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.176.212:59827/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 22:22:54,515 [I] Table x_db_version_h does not exist in database ranger
2026-08-08 22:22:54,516 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqqDyFd/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqqDyFd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.176.212:59827/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 22:22:54,747 [I] Table x_db_version_h does not exist in database ranger
2026-08-08 22:22:54,747 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-08 22:22:54,747 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqqDyFd/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqqDyFd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.176.212:59827/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskqqDyFd/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-08 22:22:54.997 UTC [26685] WARNING: there is no transaction in progress
2026-08-08 22:22:55,005 [I] create_dbversion_catalog.sql file imported successfully
2026-08-08 22:22:55,005 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqqDyFd/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqqDyFd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.176.212:59827/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 22:22:55,262 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-08 22:22:55,263 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-08 22:22:55,263 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqqDyFd/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqqDyFd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.176.212:59827/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-08 22:22:55,512 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqqDyFd/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqqDyFd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.176.212:59827/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-08 22:22:55,759 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqqDyFd/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqqDyFd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.176.212:59827/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-ncp9.c.gcp-upstream.internal','N') ;"
2026-08-08 22:22:56,007 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqqDyFd/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqqDyFd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.176.212:59827/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-08 22:22:56,264 [I] Table x_portal_user does not exist in database ranger
2026-08-08 22:22:56,264 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqqDyFd/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqqDyFd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.176.212:59827/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-08 22:22:56,535 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-08 22:22:56,536 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqqDyFd/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqqDyFd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.176.212:59827/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-08 22:22:56,786 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqqDyFd/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqqDyFd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.176.212:59827/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-08 22:22:57,042 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-08 22:22:57,042 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqqDyFd/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqqDyFd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.176.212:59827/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskqqDyFd/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-08 22:22:57.925 UTC [26844] WARNING: there is no transaction in progress
2026-08-08 22:22:57.940 UTC [26844] WARNING: there is no transaction in progress
2026-08-08 22:22:57.956 UTC [26844] WARNING: there is no transaction in progress
2026-08-08 22:22:57.969 UTC [26844] WARNING: there is no transaction in progress
2026-08-08 22:22:57.983 UTC [26844] WARNING: there is no transaction in progress
2026-08-08 22:22:57.998 UTC [26844] WARNING: there is no transaction in progress
2026-08-08 22:22:58.049 UTC [26844] WARNING: there is no transaction in progress
2026-08-08 22:22:58.055 UTC [26844] WARNING: there is no transaction in progress
2026-08-08 22:22:58.067 UTC [26844] WARNING: there is no transaction in progress
2026-08-08 22:22:58.076 UTC [26844] WARNING: there is no transaction in progress
2026-08-08 22:22:58.085 UTC [26844] WARNING: there is no transaction in progress
2026-08-08 22:22:58.097 UTC [26844] WARNING: there is no transaction in progress
2026-08-08 22:22:58.106 UTC [26844] WARNING: there is no transaction in progress
2026-08-08 22:22:58.112 UTC [26844] WARNING: there is no transaction in progress
2026-08-08 22:22:58.117 UTC [26844] WARNING: there is no transaction in progress
2026-08-08 22:22:58.123 UTC [26844] WARNING: there is no transaction in progress
2026-08-08 22:22:58,681 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-08 22:22:58,681 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqqDyFd/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqqDyFd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.176.212:59827/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-08 22:22:58,926 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-08 22:22:58,926 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqqDyFd/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqqDyFd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.176.212:59827/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-08 22:22:59,188 [I] Table x_portal_user already exists in database 'ranger'
2026-08-08 22:22:59,189 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqqDyFd/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqqDyFd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.176.212:59827/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-08 22:22:59,447 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-08 22:22:59,447 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqqDyFd/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqqDyFd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.176.212:59827/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-08 22:22:59,703 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqqDyFd/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqqDyFd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.176.212:59827/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-08 22:22:59,962 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqqDyFd/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqqDyFd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.176.212:59827/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-ncp9.c.gcp-upstream.internal';"
2026-08-08 22:23:00,193 [I] CORE_DB_SCHEMA import status has been updated
2026-08-08 22:23:00,194 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqqDyFd/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqqDyFd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.176.212:59827/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-08 22:23:00,449 [I] DB_PATCHES have already been applied
I20260808 22:23:00.454800 9923 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/minicluster-data/ranger-admin:/tmp/dist-test-taskqqDyFd/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskqqDyFd/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskqqDyFd/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqqDyFd/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260808 22:23:00.454867 9923 mini_ranger.cc:193] Using host: 127.9.176.212
I20260808 22:23:00.459033 9923 mini_ranger.cc:227] Ranger admin URL: http://127.9.176.212:41085
Aug 08, 2026 10:23:00 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 08, 2026 10:23:00 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 08, 2026 10:23:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/minicluster-data/ranger-admin/ews/webapp
Aug 08, 2026 10:23:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 08, 2026 10:23:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 08, 2026 10:23:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/minicluster-data/ranger-admin/ews/webapp].
Aug 08, 2026 10:23:01 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260808 22:23:01.689042 9923 mini_ranger.cc:160] Time spent starting Ranger: real 8.407s user 0.004s sys 0.008s
Aug 08, 2026 10:23:20 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 08, 2026 10:23:20 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.MetricsREST
Aug 08, 2026 10:23:20 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 08, 2026 10:23:20 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 08, 2026 10:23:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 08, 2026 10:23:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 08, 2026 10:23:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 08, 2026 10:23:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 08, 2026 10:23:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 08, 2026 10:23:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 08, 2026 10:23:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 08, 2026 10:23:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 08, 2026 10:23:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 08, 2026 10:23:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 08, 2026 10:23:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 08, 2026 10:23:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 08, 2026 10:23:20 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 08, 2026 10:23:21 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260808 22:23:22.038053 9923 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.9.176.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.9.176.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.9.176.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/krb5kdc/rangerkms_127.9.176.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.9.176.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/krb5kdc/rangerkms_127.9.176.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.9.176.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.9.176.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.176.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/krb5kdc/HTTP_127.9.176.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.9.176.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/krb5kdc/HTTP_127.9.176.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 08 22:23:22 dist-test-slave-ncp9 krb5kdc[26523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227802, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260808 22:23:22.123778 9923 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260808 22:23:22.133200 9923 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260808 22:23:22.879868 9923 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260808 22:23:22.879953 9923 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/minicluster-data/ranger-kms
I20260808 22:23:22.880052 9923 mini_ranger_kms.cc:227] Using postgres at 127.9.176.212:59827
2026-08-08 22:23:23,361 [I] DB FLAVOR :POSTGRES
2026-08-08 22:23:23,361 [I] --------- Verifying Ranger DB connection ---------
2026-08-08 22:23:23,361 [I] Checking connection
2026-08-08 22:23:23,361 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqqDyFd/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.176.212:59827/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-08 22:23:23,609 [I] connection success
2026-08-08 22:23:23,609 [I] --------- Verifying Ranger DB tables ---------
2026-08-08 22:23:23,609 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-08 22:23:23,610 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqqDyFd/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.176.212:59827/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-08 22:23:23,867 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-08 22:23:23,868 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-08 22:23:23,868 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-08 22:23:23,868 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqqDyFd/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.176.212:59827/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-08 22:23:24,137 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260808 22:23:24.577946 9923 mini_ranger_kms.cc:310] Created kms service
I20260808 22:23:24.753890 9923 mini_ranger_kms.cc:326] Created kudu user
I20260808 22:23:24.797514 9923 mini_ranger_kms.cc:343] Created rangerkms user
Aug 08 22:23:24 dist-test-slave-ncp9 krb5kdc[26523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227802, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.9.176.212@KRBTEST.COM
I20260808 22:23:25.162501 9923 mini_ranger_kms.cc:384] Added ranger policy
I20260808 22:23:25.162655 9923 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/minicluster-data/ranger-kms:/tmp/dist-test-taskqqDyFd/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqqDyFd/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskqqDyFd/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260808 22:23:25.162676 9923 mini_ranger_kms.cc:240] Using host: 127.9.176.212
I20260808 22:23:25.167620 9923 mini_ranger_kms.cc:276] Ranger KMS PID: 27128
I20260808 22:23:25.167721 9923 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.9.176.212:48431
Aug 08, 2026 10:23:26 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 08, 2026 10:23:26 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 08, 2026 10:23:26 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 08, 2026 10:23:26 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 08, 2026 10:23:26 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/minicluster-data/ranger-kms/ews/webapp].
Aug 08, 2026 10:23:26 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 08 22:23:27 dist-test-slave-ncp9 krb5kdc[26523](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786227807, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.176.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:23:27.667486 9923 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.544s user 0.027s sys 0.247s
I20260808 22:23:27.667619 9923 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260808 22:23:27.667738 9923 mini_ranger_kms.cc:401] 127.9.176.212:48431/kms/v1/keys
Aug 08 22:23:28 dist-test-slave-ncp9 krb5kdc[26523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227807, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.176.212@KRBTEST.COM for HTTP/127.9.176.212@KRBTEST.COM
Aug 08 22:23:28 dist-test-slave-ncp9 krb5kdc[26523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227807, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.176.212@KRBTEST.COM for HTTP/127.9.176.212@KRBTEST.COM
Aug 08 22:23:30 dist-test-slave-ncp9 krb5kdc[26523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227810, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.9.176.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.176.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.176.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab.
Entry for principal kudu/127.9.176.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab.
I20260808 22:23:30.592347 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:38861
--webserver_interface=127.9.176.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab
--principal=kudu/127.9.176.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:39329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.176.254:38861
--ranger_config_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.176.212:48431/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:23:30.704605 27201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:23:30.704841 27201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:23:30.704913 27201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:23:30.709115 27201 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:23:30.709173 27201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:23:30.709199 27201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:23:30.709230 27201 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:23:30.709259 27201 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:23:30.714648 27201 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:39329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/minicluster-data/master-0/wal
--ranger_kms_url=127.9.176.212:48431/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.9.176.254:38861
--ranger_config_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab
--principal=kudu/127.9.176.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:38861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.176.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27201
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:23:30.715590 27201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:23:30.716550 27201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:23:30.724956 27208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:23:30.725095 27201 server_base.cc:1034] running on GCE node
W20260808 22:23:30.724974 27207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:23:30.725333 27212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:23:30.726004 27201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:23:30.727368 27201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:23:30.728559 27201 hybrid_clock.cc:648] HybridClock initialized: now 1786227810728532 us; error 37 us; skew 500 ppm
Aug 08 22:23:30 dist-test-slave-ncp9 krb5kdc[26523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227810, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:23:30.735313 27201 init.cc:377] Logged in from keytab as kudu/127.9.176.254@KRBTEST.COM (short username kudu)
I20260808 22:23:30.736431 27201 webserver.cc:460] Webserver started at http://127.9.176.254:35091/ using document root <none> and password file <none>
I20260808 22:23:30.737133 27201 fs_manager.cc:359] Metadata directory not provided
I20260808 22:23:30.737210 27201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:23:30.737437 27201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 22:23:30 dist-test-slave-ncp9 krb5kdc[26523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227810, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.254@KRBTEST.COM for HTTP/127.9.176.212@KRBTEST.COM
I20260808 22:23:30.795812 27201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/minicluster-data/master-0/data/instance:
uuid: "7316be129c714498aa24cb018a9c2718"
format_stamp: "Formatted at 2026-08-08 22:23:30 on dist-test-slave-ncp9"
server_key: "c0cc635cfcb08c4b20af963b4f206e2b"
server_key_iv: "30ca24cd25c9b42d73cf533faac77162"
server_key_version: "kuduclusterkey@0"
I20260808 22:23:30.796569 27201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/minicluster-data/master-0/wal/instance:
uuid: "7316be129c714498aa24cb018a9c2718"
format_stamp: "Formatted at 2026-08-08 22:23:30 on dist-test-slave-ncp9"
server_key: "c0cc635cfcb08c4b20af963b4f206e2b"
server_key_iv: "30ca24cd25c9b42d73cf533faac77162"
server_key_version: "kuduclusterkey@0"
I20260808 22:23:30.800920 27201 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260808 22:23:30.818571 27228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:23:30.820032 27201 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 22:23:30.820168 27201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/minicluster-data/master-0/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/minicluster-data/master-0/wal
uuid: "7316be129c714498aa24cb018a9c2718"
format_stamp: "Formatted at 2026-08-08 22:23:30 on dist-test-slave-ncp9"
server_key: "c0cc635cfcb08c4b20af963b4f206e2b"
server_key_iv: "30ca24cd25c9b42d73cf533faac77162"
server_key_version: "kuduclusterkey@0"
I20260808 22:23:30.820283 27201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:23:30.834555 27201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:23:30.837342 27201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:23:30.837517 27201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:23:30.845697 27201 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.254:38861
I20260808 22:23:30.845839 27316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.254:38861 every 8 connection(s)
I20260808 22:23:30.846683 27201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
I20260808 22:23:30.849478 27317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:23:30.849653 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 27201
I20260808 22:23:30.849779 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/minicluster-data/master-0/wal/instance
I20260808 22:23:30.850114 9923 external_mini_cluster.cc:1428] Setting key eae64976d69aa6610a85bc11650a4401
I20260808 22:23:30.856655 27317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7316be129c714498aa24cb018a9c2718: Bootstrap starting.
I20260808 22:23:30.859115 27317 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7316be129c714498aa24cb018a9c2718: Neither blocks nor log segments found. Creating new log.
I20260808 22:23:30.859814 27317 log.cc:824] T 00000000000000000000000000000000 P 7316be129c714498aa24cb018a9c2718: Log is configured to *not* fsync() on all Append() calls
Aug 08 22:23:30 dist-test-slave-ncp9 krb5kdc[26523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227810, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
I20260808 22:23:30.861933 27317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7316be129c714498aa24cb018a9c2718: No bootstrap required, opened a new log
I20260808 22:23:30.865056 27317 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7316be129c714498aa24cb018a9c2718 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7316be129c714498aa24cb018a9c2718" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 38861 } }
I20260808 22:23:30.865273 27317 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7316be129c714498aa24cb018a9c2718 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:23:30.865345 27317 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7316be129c714498aa24cb018a9c2718 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7316be129c714498aa24cb018a9c2718, State: Initialized, Role: FOLLOWER
I20260808 22:23:30.865850 27317 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7316be129c714498aa24cb018a9c2718 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7316be129c714498aa24cb018a9c2718" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 38861 } }
I20260808 22:23:30.866042 27317 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7316be129c714498aa24cb018a9c2718 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:23:30.866148 27317 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7316be129c714498aa24cb018a9c2718 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:23:30.866263 27317 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7316be129c714498aa24cb018a9c2718 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:23:30.867039 27317 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7316be129c714498aa24cb018a9c2718 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7316be129c714498aa24cb018a9c2718" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 38861 } }
I20260808 22:23:30.867107 27321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:23:30.851637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40630 (local address 127.9.176.254:38861)
0808 22:23:30.852151 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0808 22:23:30.852169 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0808 22:23:30.852215 (+ 46us) server_negotiation.cc:407] Connection header received
0808 22:23:30.852990 (+ 775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:23:30.853016 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:23:30.853429 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:23:30.853801 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:23:30.854678 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:23:30.856129 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:23:30.857390 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:23:30.858076 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:23:30.858224 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:23:30.861106 (+ 2882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:23:30.861112 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:23:30.861130 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:23:30.861161 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:23:30.863908 (+ 2747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:23:30.864400 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:23:30.864405 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:23:30.864411 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:23:30.864542 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:23:30.864831 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:23:30.864833 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:23:30.864835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:23:30.865369 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:23:30.865616 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0808 22:23:30.865907 (+ 291us) server_negotiation.cc:299] Negotiation successful
0808 22:23:30.866192 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":158,"threads_started":1}
I20260808 22:23:30.867388 27317 leader_election.cc:302] T 00000000000000000000000000000000 P 7316be129c714498aa24cb018a9c2718 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7316be129c714498aa24cb018a9c2718; no voters:
I20260808 22:23:30.867709 27317 leader_election.cc:288] T 00000000000000000000000000000000 P 7316be129c714498aa24cb018a9c2718 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:23:30.867790 27324 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7316be129c714498aa24cb018a9c2718 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:23:30.868031 27324 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7316be129c714498aa24cb018a9c2718 [term 1 LEADER]: Becoming Leader. State: Replica: 7316be129c714498aa24cb018a9c2718, State: Running, Role: LEADER
I20260808 22:23:30.868364 27324 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7316be129c714498aa24cb018a9c2718 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7316be129c714498aa24cb018a9c2718" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 38861 } }
I20260808 22:23:30.869064 27317 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7316be129c714498aa24cb018a9c2718 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:23:30.869606 27324 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7316be129c714498aa24cb018a9c2718 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7316be129c714498aa24cb018a9c2718. Latest consensus state: current_term: 1 leader_uuid: "7316be129c714498aa24cb018a9c2718" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7316be129c714498aa24cb018a9c2718" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 38861 } } }
I20260808 22:23:30.869796 27324 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7316be129c714498aa24cb018a9c2718 [sys.catalog]: This master's current role is: LEADER
I20260808 22:23:30.870042 27324 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7316be129c714498aa24cb018a9c2718 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7316be129c714498aa24cb018a9c2718" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7316be129c714498aa24cb018a9c2718" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 38861 } } }
I20260808 22:23:30.870178 27324 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7316be129c714498aa24cb018a9c2718 [sys.catalog]: This master's current role is: LEADER
I20260808 22:23:30.871379 27317 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260808 22:23:30.875097 27334 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:23:30.879581 27334 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:23:30.886871 27334 catalog_manager.cc:1347] Generated new cluster ID: 86c00e9ff0e146b984e6d96724b17048
I20260808 22:23:30.886929 27334 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:23:30.904548 27334 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:23:30.906008 27334 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:23:30.914348 27334 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7316be129c714498aa24cb018a9c2718: Generated new TSK 0
I20260808 22:23:30.915337 27334 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.176.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.176.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.176.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.193.keytab.
Entry for principal kudu/127.9.176.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.193.keytab.
I20260808 22:23:31.669839 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.176.212:48431/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.193:0
--local_ip_for_outbound_sockets=127.9.176.193
--webserver_interface=127.9.176.193
--webserver_port=0
--tserver_master_addrs=127.9.176.254:38861
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.193.keytab
--principal=kudu/127.9.176.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:39329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:23:31.824885 27392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:23:31.825188 27392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:23:31.825249 27392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:23:31.829355 27392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:23:31.829407 27392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:23:31.829481 27392 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.193
I20260808 22:23:31.835111 27392 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:39329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.9.176.212:48431/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.193.keytab
--principal=kudu/127.9.176.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.176.193
--webserver_port=0
--tserver_master_addrs=127.9.176.254:38861
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27392
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.193
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:23:31.836097 27392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:23:31.837121 27392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:23:31.848042 27400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:23:31.848096 27404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:23:31.848491 27401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:23:31.848821 27392 server_base.cc:1034] running on GCE node
I20260808 22:23:31.849238 27392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:23:31.849908 27392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:23:31.851137 27392 hybrid_clock.cc:648] HybridClock initialized: now 1786227811851107 us; error 46 us; skew 500 ppm
Aug 08 22:23:31 dist-test-slave-ncp9 krb5kdc[26523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227811, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:23:31.856310 27392 init.cc:377] Logged in from keytab as kudu/127.9.176.193@KRBTEST.COM (short username kudu)
I20260808 22:23:31.857362 27392 webserver.cc:460] Webserver started at http://127.9.176.193:41091/ using document root <none> and password file <none>
I20260808 22:23:31.858053 27392 fs_manager.cc:359] Metadata directory not provided
I20260808 22:23:31.858194 27392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:23:31.858506 27392 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 22:23:31 dist-test-slave-ncp9 krb5kdc[26523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227811, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.193@KRBTEST.COM for HTTP/127.9.176.212@KRBTEST.COM
I20260808 22:23:31.877748 27392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/minicluster-data/ts-0/data/instance:
uuid: "6829c2aa81ae4ec9ace4283d33d96159"
format_stamp: "Formatted at 2026-08-08 22:23:31 on dist-test-slave-ncp9"
server_key: "8bb4ef244508131bff5818af4e3ac452"
server_key_iv: "f09aaf50639f6281bd7d091a27fe4abd"
server_key_version: "kuduclusterkey@0"
I20260808 22:23:31.878526 27392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/minicluster-data/ts-0/wal/instance:
uuid: "6829c2aa81ae4ec9ace4283d33d96159"
format_stamp: "Formatted at 2026-08-08 22:23:31 on dist-test-slave-ncp9"
server_key: "8bb4ef244508131bff5818af4e3ac452"
server_key_iv: "f09aaf50639f6281bd7d091a27fe4abd"
server_key_version: "kuduclusterkey@0"
I20260808 22:23:31.894307 27392 fs_manager.cc:691] Time spent creating directory manager: real 0.015s user 0.003s sys 0.000s
I20260808 22:23:31.916383 27417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:23:31.917773 27392 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.004s sys 0.000s
I20260808 22:23:31.917884 27392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/minicluster-data/ts-0/wal
uuid: "6829c2aa81ae4ec9ace4283d33d96159"
format_stamp: "Formatted at 2026-08-08 22:23:31 on dist-test-slave-ncp9"
server_key: "8bb4ef244508131bff5818af4e3ac452"
server_key_iv: "f09aaf50639f6281bd7d091a27fe4abd"
server_key_version: "kuduclusterkey@0"
I20260808 22:23:31.918021 27392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:23:31.944649 27392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:23:31.948325 27392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:23:31.948544 27392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:23:31.949332 27392 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:23:31.950438 27392 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:23:31.950505 27392 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:23:31.950578 27392 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:23:31.950608 27392 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:23:31.964370 27392 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.193:45989
I20260808 22:23:31.965116 27593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.193:45989 every 8 connection(s)
I20260808 22:23:31.965528 27392 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
I20260808 22:23:31.973706 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 27392
I20260808 22:23:31.973836 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/minicluster-data/ts-0/wal/instance
I20260808 22:23:31.974123 9923 external_mini_cluster.cc:1428] Setting key a19ec50e6f223931d57232856410ee78
Aug 08 22:23:31 dist-test-slave-ncp9 krb5kdc[26523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227811, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.193@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
I20260808 22:23:31.992115 27599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:23:31.969418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.193:56819 (local address 127.9.176.254:38861)
0808 22:23:31.971502 (+ 2084us) server_negotiation.cc:206] Beginning negotiation
0808 22:23:31.971506 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:23:31.971520 (+ 14us) server_negotiation.cc:407] Connection header received
0808 22:23:31.971576 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:23:31.971578 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:23:31.971625 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:23:31.971705 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:23:31.977897 (+ 6192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:23:31.978772 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:23:31.980987 (+ 2215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:23:31.981341 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:23:31.981427 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:23:31.987736 (+ 6309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:23:31.987740 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:23:31.987743 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:23:31.987770 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:23:31.989603 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:23:31.990241 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:23:31.990245 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:23:31.990247 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:23:31.990345 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:23:31.990793 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:23:31.990796 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:23:31.990798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:23:31.991024 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:23:31.991108 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0808 22:23:31.991781 (+ 673us) server_negotiation.cc:299] Negotiation successful
0808 22:23:31.991916 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2001,"thread_start_us":62,"threads_started":1}
I20260808 22:23:31.993986 27598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:23:31.968238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:38861 (local address 127.9.176.193:56819)
0808 22:23:31.969886 (+ 1648us) negotiation.cc:107] Waiting for socket to connect
0808 22:23:31.969944 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0808 22:23:31.971091 (+ 1147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:23:31.972006 (+ 915us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:23:31.972014 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:23:31.972398 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:23:31.973283 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:23:31.973305 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:23:31.978951 (+ 5646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:23:31.978955 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:23:31.980428 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:23:31.980444 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:23:31.981653 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:23:31.981658 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:23:31.981890 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:23:31.985765 (+ 3875us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:23:31.985785 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:23:31.987527 (+ 1742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:23:31.989750 (+ 2223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:23:31.989756 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:23:31.989771 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:23:31.990115 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:23:31.990480 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:23:31.990483 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:23:31.990485 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:23:31.990675 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:23:31.991179 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:23:31.991185 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:23:31.991485 (+ 300us) client_negotiation.cc:769] Sending connection context
0808 22:23:31.992561 (+ 1076us) client_negotiation.cc:240] Negotiation successful
0808 22:23:31.992920 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1401,"thread_start_us":187,"threads_started":1}
I20260808 22:23:31.995335 27594 heartbeater.cc:347] Connected to a master server at 127.9.176.254:38861
I20260808 22:23:31.995605 27594 heartbeater.cc:464] Registering TS with master...
I20260808 22:23:31.996238 27594 heartbeater.cc:511] Master 127.9.176.254:38861 requested a full tablet report, sending...
I20260808 22:23:31.998098 27257 ts_manager.cc:194] Registered new tserver with Master: 6829c2aa81ae4ec9ace4283d33d96159 (127.9.176.193:45989)
WARNING: no policy specified for kudu/127.9.176.194@KRBTEST.COM; defaulting to no policy
I20260808 22:23:31.999689 27257 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.176.193@KRBTEST.COM'} at 127.9.176.193:56819
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.176.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.176.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.194.keytab.
Entry for principal kudu/127.9.176.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.194.keytab.
I20260808 22:23:32.027858 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.176.212:48431/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.194:0
--local_ip_for_outbound_sockets=127.9.176.194
--webserver_interface=127.9.176.194
--webserver_port=0
--tserver_master_addrs=127.9.176.254:38861
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.194.keytab
--principal=kudu/127.9.176.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:39329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 22:23:32 dist-test-slave-ncp9 krb5kdc[26523](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786227812, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260808 22:23:32.151091 27610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:23:32.151340 27610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:23:32.151396 27610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:23:32.155970 27610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:23:32.156100 27610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:23:32.156196 27610 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.194
I20260808 22:23:32.161806 27610 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:39329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.9.176.212:48431/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.194.keytab
--principal=kudu/127.9.176.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.176.194
--webserver_port=0
--tserver_master_addrs=127.9.176.254:38861
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27610
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.194
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:23:32.163054 27610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:23:32.164245 27610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:23:32.174108 27623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:23:32.174049 27620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:23:32.174563 27619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:23:32.175108 27610 server_base.cc:1034] running on GCE node
I20260808 22:23:32.175478 27610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:23:32.176092 27610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:23:32.177333 27610 hybrid_clock.cc:648] HybridClock initialized: now 1786227812177305 us; error 35 us; skew 500 ppm
Aug 08 22:23:32 dist-test-slave-ncp9 krb5kdc[26523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227812, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:23:32.180123 27610 init.cc:377] Logged in from keytab as kudu/127.9.176.194@KRBTEST.COM (short username kudu)
I20260808 22:23:32.180941 27610 webserver.cc:460] Webserver started at http://127.9.176.194:39225/ using document root <none> and password file <none>
I20260808 22:23:32.181547 27610 fs_manager.cc:359] Metadata directory not provided
I20260808 22:23:32.181615 27610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:23:32.181828 27610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 22:23:32 dist-test-slave-ncp9 krb5kdc[26523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227812, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.194@KRBTEST.COM for HTTP/127.9.176.212@KRBTEST.COM
I20260808 22:23:32.201578 27610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/minicluster-data/ts-1/data/instance:
uuid: "9ca6543814fb43abbf580a87de0b53f9"
format_stamp: "Formatted at 2026-08-08 22:23:32 on dist-test-slave-ncp9"
server_key: "0ed312b0c1cdfbe8f4aef6d5e32a404d"
server_key_iv: "d284d9d1bcad60aac5f7995cbeaa24e2"
server_key_version: "kuduclusterkey@0"
I20260808 22:23:32.202342 27610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/minicluster-data/ts-1/wal/instance:
uuid: "9ca6543814fb43abbf580a87de0b53f9"
format_stamp: "Formatted at 2026-08-08 22:23:32 on dist-test-slave-ncp9"
server_key: "0ed312b0c1cdfbe8f4aef6d5e32a404d"
server_key_iv: "d284d9d1bcad60aac5f7995cbeaa24e2"
server_key_version: "kuduclusterkey@0"
I20260808 22:23:32.207799 27610 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 22:23:32.231127 27631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:23:32.232882 27610 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260808 22:23:32.233057 27610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/minicluster-data/ts-1/wal
uuid: "9ca6543814fb43abbf580a87de0b53f9"
format_stamp: "Formatted at 2026-08-08 22:23:32 on dist-test-slave-ncp9"
server_key: "0ed312b0c1cdfbe8f4aef6d5e32a404d"
server_key_iv: "d284d9d1bcad60aac5f7995cbeaa24e2"
server_key_version: "kuduclusterkey@0"
I20260808 22:23:32.233201 27610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:23:32.267911 27610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:23:32.271994 27610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:23:32.272262 27610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:23:32.273105 27610 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:23:32.274322 27610 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:23:32.274430 27610 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:23:32.274551 27610 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:23:32.274622 27610 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:23:32.294620 27610 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.194:44251
I20260808 22:23:32.296245 27610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
I20260808 22:23:32.297333 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 27610
I20260808 22:23:32.297470 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/minicluster-data/ts-1/wal/instance
I20260808 22:23:32.297806 9923 external_mini_cluster.cc:1428] Setting key 24f9389aebe7d1c2de84dcffc9006a67
I20260808 22:23:32.305490 27824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.194:44251 every 8 connection(s)
Aug 08 22:23:32 dist-test-slave-ncp9 krb5kdc[26523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227812, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.194@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
I20260808 22:23:32.328548 27599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:23:32.309147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.194:54649 (local address 127.9.176.254:38861)
0808 22:23:32.309314 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0808 22:23:32.309320 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 22:23:32.313522 (+ 4202us) server_negotiation.cc:407] Connection header received
0808 22:23:32.315187 (+ 1665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:23:32.315191 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:23:32.315248 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:23:32.315301 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:23:32.317366 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:23:32.318217 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:23:32.320206 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:23:32.320549 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:23:32.320627 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:23:32.324238 (+ 3611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:23:32.324242 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:23:32.324245 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:23:32.324274 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:23:32.325984 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:23:32.326613 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:23:32.326618 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:23:32.326620 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:23:32.326740 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:23:32.327162 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:23:32.327165 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:23:32.327167 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:23:32.327390 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:23:32.327473 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0808 22:23:32.328212 (+ 739us) server_negotiation.cc:299] Negotiation successful
0808 22:23:32.328346 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 22:23:32.330535 27831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:23:32.308413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:38861 (local address 127.9.176.194:54649)
0808 22:23:32.313352 (+ 4939us) negotiation.cc:107] Waiting for socket to connect
0808 22:23:32.313413 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0808 22:23:32.314919 (+ 1506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:23:32.315756 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:23:32.315766 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:23:32.316389 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:23:32.317186 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:23:32.317210 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:23:32.318386 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:23:32.318390 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:23:32.320062 (+ 1672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:23:32.320074 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:23:32.320883 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:23:32.320888 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:23:32.321225 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:23:32.321905 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:23:32.321929 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:23:32.324037 (+ 2108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:23:32.326114 (+ 2077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:23:32.326120 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:23:32.326136 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:23:32.326485 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:23:32.326865 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:23:32.326868 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:23:32.326869 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:23:32.327049 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:23:32.327547 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:23:32.327554 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:23:32.327900 (+ 346us) client_negotiation.cc:769] Sending connection context
0808 22:23:32.329063 (+ 1163us) client_negotiation.cc:240] Negotiation successful
0808 22:23:32.329412 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4652,"thread_start_us":202,"threads_started":1}
I20260808 22:23:32.332196 27825 heartbeater.cc:347] Connected to a master server at 127.9.176.254:38861
I20260808 22:23:32.332448 27825 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.9.176.195@KRBTEST.COM; defaulting to no policy
I20260808 22:23:32.333113 27825 heartbeater.cc:511] Master 127.9.176.254:38861 requested a full tablet report, sending...
I20260808 22:23:32.334403 27257 ts_manager.cc:194] Registered new tserver with Master: 9ca6543814fb43abbf580a87de0b53f9 (127.9.176.194:44251)
I20260808 22:23:32.335237 27257 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.176.194@KRBTEST.COM'} at 127.9.176.194:54649
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.176.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.176.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.195.keytab.
Entry for principal kudu/127.9.176.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.195.keytab.
I20260808 22:23:32.364400 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.176.212:48431/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.195:0
--local_ip_for_outbound_sockets=127.9.176.195
--webserver_interface=127.9.176.195
--webserver_port=0
--tserver_master_addrs=127.9.176.254:38861
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.195.keytab
--principal=kudu/127.9.176.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:39329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:23:32.574812 27833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:23:32.575634 27833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:23:32.575778 27833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:23:32.581631 27833 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:23:32.581827 27833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:23:32.581976 27833 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.195
I20260808 22:23:32.589910 27833 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:39329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.9.176.212:48431/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.195.keytab
--principal=kudu/127.9.176.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.176.195
--webserver_port=0
--tserver_master_addrs=127.9.176.254:38861
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27833
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.195
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:23:32.591416 27833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:23:32.592795 27833 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:23:32.608179 27839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:23:32.608157 27840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:23:32.609254 27833 server_base.cc:1034] running on GCE node
W20260808 22:23:32.609455 27842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:23:32.610038 27833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:23:32.610880 27833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:23:32.612277 27833 hybrid_clock.cc:648] HybridClock initialized: now 1786227812612200 us; error 95 us; skew 500 ppm
Aug 08 22:23:32 dist-test-slave-ncp9 krb5kdc[26523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227812, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:23:32.617085 27833 init.cc:377] Logged in from keytab as kudu/127.9.176.195@KRBTEST.COM (short username kudu)
I20260808 22:23:32.618324 27833 webserver.cc:460] Webserver started at http://127.9.176.195:45909/ using document root <none> and password file <none>
I20260808 22:23:32.619138 27833 fs_manager.cc:359] Metadata directory not provided
I20260808 22:23:32.619299 27833 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:23:32.619634 27833 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 22:23:32 dist-test-slave-ncp9 krb5kdc[26523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227812, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.195@KRBTEST.COM for HTTP/127.9.176.212@KRBTEST.COM
I20260808 22:23:32.643595 27833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/minicluster-data/ts-2/data/instance:
uuid: "6cc3bb0ea33149fe96a5170c5e74cc59"
format_stamp: "Formatted at 2026-08-08 22:23:32 on dist-test-slave-ncp9"
server_key: "9e06bc7e922de3590260e199b8546c3e"
server_key_iv: "397c07452141ec4b3ec2cda76a80605e"
server_key_version: "kuduclusterkey@0"
I20260808 22:23:32.644589 27833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/minicluster-data/ts-2/wal/instance:
uuid: "6cc3bb0ea33149fe96a5170c5e74cc59"
format_stamp: "Formatted at 2026-08-08 22:23:32 on dist-test-slave-ncp9"
server_key: "9e06bc7e922de3590260e199b8546c3e"
server_key_iv: "397c07452141ec4b3ec2cda76a80605e"
server_key_version: "kuduclusterkey@0"
I20260808 22:23:32.650605 27833 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260808 22:23:32.674340 27851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:23:32.676029 27833 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 22:23:32.676169 27833 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/minicluster-data/ts-2/wal
uuid: "6cc3bb0ea33149fe96a5170c5e74cc59"
format_stamp: "Formatted at 2026-08-08 22:23:32 on dist-test-slave-ncp9"
server_key: "9e06bc7e922de3590260e199b8546c3e"
server_key_iv: "397c07452141ec4b3ec2cda76a80605e"
server_key_version: "kuduclusterkey@0"
I20260808 22:23:32.676295 27833 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Aug 08 22:23:32 dist-test-slave-ncp9 krb5kdc[26523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227812, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.254@KRBTEST.COM for HTTP/127.9.176.212@KRBTEST.COM
I20260808 22:23:32.699684 27833 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:23:32.703541 27833 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:23:32.703936 27833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:23:32.704774 27833 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:23:32.706041 27833 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:23:32.706156 27833 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:23:32.706271 27833 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:23:32.706341 27833 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:23:32.731423 27833 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.195:40055
I20260808 22:23:32.732843 27833 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
I20260808 22:23:32.735991 28045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.195:40055 every 8 connection(s)
I20260808 22:23:32.736253 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 27833
I20260808 22:23:32.736397 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/minicluster-data/ts-2/wal/instance
I20260808 22:23:32.736734 9923 external_mini_cluster.cc:1428] Setting key b42c9654b807c973284acbb3927e4614
Aug 08 22:23:32 dist-test-slave-ncp9 krb5kdc[26523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227812, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.195@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
I20260808 22:23:32.794447 27599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:23:32.740180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.195:38111 (local address 127.9.176.254:38861)
0808 22:23:32.740297 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0808 22:23:32.740302 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:23:32.749529 (+ 9227us) server_negotiation.cc:407] Connection header received
0808 22:23:32.751145 (+ 1616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:23:32.751149 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:23:32.751201 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:23:32.751279 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:23:32.753310 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:23:32.754160 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:23:32.755848 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:23:32.756167 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:23:32.756245 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:23:32.781738 (+ 25493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:23:32.781743 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:23:32.781747 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:23:32.781790 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:23:32.783771 (+ 1981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:23:32.785173 (+ 1402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:23:32.785178 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:23:32.785180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:23:32.785295 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:23:32.787986 (+ 2691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:23:32.787991 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:23:32.792682 (+ 4691us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:23:32.793042 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:23:32.793156 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0808 22:23:32.794072 (+ 916us) server_negotiation.cc:299] Negotiation successful
0808 22:23:32.794228 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260808 22:23:32.796257 28049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:23:32.738958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:38861 (local address 127.9.176.195:38111)
0808 22:23:32.749345 (+ 10387us) negotiation.cc:107] Waiting for socket to connect
0808 22:23:32.749414 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0808 22:23:32.750878 (+ 1464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:23:32.751761 (+ 883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:23:32.751771 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:23:32.752302 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:23:32.753145 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:23:32.753168 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:23:32.754323 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:23:32.754326 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:23:32.755718 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:23:32.755729 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:23:32.756452 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:23:32.756456 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:23:32.756662 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:23:32.757492 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:23:32.757513 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:23:32.760576 (+ 3063us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:23:32.783961 (+ 23385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:23:32.783969 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:23:32.783990 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:23:32.784449 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:23:32.787574 (+ 3125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:23:32.787577 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:23:32.787579 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:23:32.787839 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:23:32.793256 (+ 5417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:23:32.793265 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:23:32.793716 (+ 451us) client_negotiation.cc:769] Sending connection context
0808 22:23:32.794663 (+ 947us) client_negotiation.cc:240] Negotiation successful
0808 22:23:32.795085 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10110,"thread_start_us":197,"threads_started":1}
I20260808 22:23:32.798075 28046 heartbeater.cc:347] Connected to a master server at 127.9.176.254:38861
I20260808 22:23:32.798323 28046 heartbeater.cc:464] Registering TS with master...
I20260808 22:23:32.798889 28046 heartbeater.cc:511] Master 127.9.176.254:38861 requested a full tablet report, sending...
I20260808 22:23:32.800277 27257 ts_manager.cc:194] Registered new tserver with Master: 6cc3bb0ea33149fe96a5170c5e74cc59 (127.9.176.195:40055)
I20260808 22:23:32.801044 27257 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.176.195@KRBTEST.COM'} at 127.9.176.195:38111
I20260808 22:23:32.811089 9923 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 22:23:32.824270 27599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:23:32.813597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40634 (local address 127.9.176.254:38861)
0808 22:23:32.813726 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0808 22:23:32.813730 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:23:32.814678 (+ 948us) server_negotiation.cc:407] Connection header received
0808 22:23:32.814874 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:23:32.814877 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:23:32.814923 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:23:32.814972 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:23:32.816052 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:23:32.816884 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:23:32.818140 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:23:32.818486 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:23:32.818559 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:23:32.820393 (+ 1834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:23:32.820397 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:23:32.820400 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:23:32.820424 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:23:32.822109 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:23:32.822696 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:23:32.822699 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:23:32.822701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:23:32.822789 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:23:32.823107 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:23:32.823111 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:23:32.823113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:23:32.823363 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:23:32.823465 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0808 22:23:32.823977 (+ 512us) server_negotiation.cc:299] Negotiation successful
0808 22:23:32.824111 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 22:23:32.828023 27257 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40634:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 22:23:32.832473 27257 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 22:23:32.863752 28066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:23:32.846488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.194:44251 (local address 127.0.0.1:33948)
0808 22:23:32.851784 (+ 5296us) negotiation.cc:107] Waiting for socket to connect
0808 22:23:32.851797 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 22:23:32.851885 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:23:32.852836 (+ 951us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:23:32.852838 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:23:32.852853 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:23:32.852954 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:23:32.852959 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:23:32.854354 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:23:32.854357 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:23:32.855707 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:23:32.855716 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:23:32.863202 (+ 7486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:23:32.863206 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:23:32.863319 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:23:32.863365 (+ 46us) client_negotiation.cc:769] Sending connection context
0808 22:23:32.863471 (+ 106us) client_negotiation.cc:240] Negotiation successful
0808 22:23:32.863611 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5238,"thread_start_us":137,"threads_started":1}
I20260808 22:23:32.864370 28069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:23:32.846992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33948 (local address 127.9.176.194:44251)
0808 22:23:32.850070 (+ 3078us) server_negotiation.cc:206] Beginning negotiation
0808 22:23:32.850077 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 22:23:32.851980 (+ 1903us) server_negotiation.cc:407] Connection header received
0808 22:23:32.852061 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:23:32.852066 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:23:32.852259 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:23:32.852381 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:23:32.853058 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:23:32.854235 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:23:32.860540 (+ 6305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:23:32.861561 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:23:32.861670 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:23:32.861815 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0808 22:23:32.864126 (+ 2311us) server_negotiation.cc:299] Negotiation successful
0808 22:23:32.864237 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2966,"thread_start_us":160,"threads_started":1}
I20260808 22:23:32.864499 28067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:23:32.845301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53704 (local address 127.9.176.193:45989)
0808 22:23:32.851045 (+ 5744us) server_negotiation.cc:206] Beginning negotiation
0808 22:23:32.851051 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 22:23:32.851072 (+ 21us) server_negotiation.cc:407] Connection header received
0808 22:23:32.851154 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:23:32.851160 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:23:32.851359 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:23:32.851502 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:23:32.856466 (+ 4964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:23:32.857659 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:23:32.861874 (+ 4215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:23:32.862752 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:23:32.863601 (+ 849us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:23:32.863744 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0808 22:23:32.864262 (+ 518us) server_negotiation.cc:299] Negotiation successful
0808 22:23:32.864354 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5643,"thread_start_us":68,"threads_started":1}
I20260808 22:23:32.864169 28070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:23:32.844155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.193:45989 (local address 127.0.0.1:53704)
0808 22:23:32.850583 (+ 6428us) negotiation.cc:107] Waiting for socket to connect
0808 22:23:32.850629 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0808 22:23:32.850777 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:23:32.852531 (+ 1754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:23:32.852538 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:23:32.852558 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:23:32.852735 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:23:32.852742 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:23:32.857780 (+ 5038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:23:32.857784 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:23:32.858882 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:23:32.858891 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:23:32.863797 (+ 4906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:23:32.863800 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:23:32.863898 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:23:32.863920 (+ 22us) client_negotiation.cc:769] Sending connection context
0808 22:23:32.863982 (+ 62us) client_negotiation.cc:240] Negotiation successful
0808 22:23:32.864051 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6226,"thread_start_us":129,"threads_started":1}
I20260808 22:23:32.868002 27705 tablet_service.cc:1467] Processing CreateTablet for tablet 7551fd743f334ce796ed4008170aec44 (DEFAULT_TABLE table=test-table [id=f7043367cab847aba2238fdbb68d8122]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:23:32.867996 27493 tablet_service.cc:1467] Processing CreateTablet for tablet 7551fd743f334ce796ed4008170aec44 (DEFAULT_TABLE table=test-table [id=f7043367cab847aba2238fdbb68d8122]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:23:32.869185 27493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7551fd743f334ce796ed4008170aec44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:23:32.869185 27705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7551fd743f334ce796ed4008170aec44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:23:32.877419 28068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:23:32.847772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.195:40055 (local address 127.0.0.1:53962)
0808 22:23:32.855288 (+ 7516us) negotiation.cc:107] Waiting for socket to connect
0808 22:23:32.855300 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 22:23:32.855385 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:23:32.858995 (+ 3610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:23:32.858998 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:23:32.859014 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:23:32.859153 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:23:32.859158 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:23:32.860602 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:23:32.860605 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:23:32.861846 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:23:32.861857 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:23:32.877022 (+ 15165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:23:32.877026 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:23:32.877135 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:23:32.877160 (+ 25us) client_negotiation.cc:769] Sending connection context
0808 22:23:32.877223 (+ 63us) client_negotiation.cc:240] Negotiation successful
0808 22:23:32.877292 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7460,"thread_start_us":144,"threads_started":1}
I20260808 22:23:32.879482 28072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:23:32.848273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53962 (local address 127.9.176.195:40055)
0808 22:23:32.850263 (+ 1990us) server_negotiation.cc:206] Beginning negotiation
0808 22:23:32.850318 (+ 55us) server_negotiation.cc:399] Waiting for connection header
0808 22:23:32.855857 (+ 5539us) server_negotiation.cc:407] Connection header received
0808 22:23:32.855940 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:23:32.855945 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:23:32.856117 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:23:32.856310 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:23:32.859278 (+ 2968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:23:32.860481 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:23:32.862051 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:23:32.862893 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:23:32.862989 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:23:32.863145 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0808 22:23:32.879174 (+ 16029us) server_negotiation.cc:299] Negotiation successful
0808 22:23:32.879295 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1899,"thread_start_us":176,"threads_started":1}
I20260808 22:23:32.880659 28073 tablet_bootstrap.cc:492] T 7551fd743f334ce796ed4008170aec44 P 9ca6543814fb43abbf580a87de0b53f9: Bootstrap starting.
I20260808 22:23:32.882781 27932 tablet_service.cc:1467] Processing CreateTablet for tablet 7551fd743f334ce796ed4008170aec44 (DEFAULT_TABLE table=test-table [id=f7043367cab847aba2238fdbb68d8122]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:23:32.883656 28073 tablet_bootstrap.cc:654] T 7551fd743f334ce796ed4008170aec44 P 9ca6543814fb43abbf580a87de0b53f9: Neither blocks nor log segments found. Creating new log.
I20260808 22:23:32.883824 27932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7551fd743f334ce796ed4008170aec44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:23:32.885007 28073 log.cc:824] T 7551fd743f334ce796ed4008170aec44 P 9ca6543814fb43abbf580a87de0b53f9: Log is configured to *not* fsync() on all Append() calls
I20260808 22:23:32.885783 28074 tablet_bootstrap.cc:492] T 7551fd743f334ce796ed4008170aec44 P 6829c2aa81ae4ec9ace4283d33d96159: Bootstrap starting.
I20260808 22:23:32.887385 28073 tablet_bootstrap.cc:492] T 7551fd743f334ce796ed4008170aec44 P 9ca6543814fb43abbf580a87de0b53f9: No bootstrap required, opened a new log
I20260808 22:23:32.887585 28073 ts_tablet_manager.cc:1397] T 7551fd743f334ce796ed4008170aec44 P 9ca6543814fb43abbf580a87de0b53f9: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260808 22:23:32.888435 28074 tablet_bootstrap.cc:654] T 7551fd743f334ce796ed4008170aec44 P 6829c2aa81ae4ec9ace4283d33d96159: Neither blocks nor log segments found. Creating new log.
I20260808 22:23:32.889672 28074 log.cc:824] T 7551fd743f334ce796ed4008170aec44 P 6829c2aa81ae4ec9ace4283d33d96159: Log is configured to *not* fsync() on all Append() calls
I20260808 22:23:32.891798 28074 tablet_bootstrap.cc:492] T 7551fd743f334ce796ed4008170aec44 P 6829c2aa81ae4ec9ace4283d33d96159: No bootstrap required, opened a new log
I20260808 22:23:32.891978 28074 ts_tablet_manager.cc:1397] T 7551fd743f334ce796ed4008170aec44 P 6829c2aa81ae4ec9ace4283d33d96159: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.006s
I20260808 22:23:32.893464 28078 tablet_bootstrap.cc:492] T 7551fd743f334ce796ed4008170aec44 P 6cc3bb0ea33149fe96a5170c5e74cc59: Bootstrap starting.
I20260808 22:23:32.893463 28073 raft_consensus.cc:348] T 7551fd743f334ce796ed4008170aec44 P 9ca6543814fb43abbf580a87de0b53f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6829c2aa81ae4ec9ace4283d33d96159" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 45989 } } peers { permanent_uuid: "9ca6543814fb43abbf580a87de0b53f9" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 44251 } } peers { permanent_uuid: "6cc3bb0ea33149fe96a5170c5e74cc59" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40055 } }
I20260808 22:23:32.893715 28073 raft_consensus.cc:374] T 7551fd743f334ce796ed4008170aec44 P 9ca6543814fb43abbf580a87de0b53f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:23:32.893766 28073 raft_consensus.cc:729] T 7551fd743f334ce796ed4008170aec44 P 9ca6543814fb43abbf580a87de0b53f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ca6543814fb43abbf580a87de0b53f9, State: Initialized, Role: FOLLOWER
I20260808 22:23:32.894250 28073 consensus_queue.cc:260] T 7551fd743f334ce796ed4008170aec44 P 9ca6543814fb43abbf580a87de0b53f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6829c2aa81ae4ec9ace4283d33d96159" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 45989 } } peers { permanent_uuid: "9ca6543814fb43abbf580a87de0b53f9" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 44251 } } peers { permanent_uuid: "6cc3bb0ea33149fe96a5170c5e74cc59" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40055 } }
I20260808 22:23:32.896030 28073 ts_tablet_manager.cc:1428] T 7551fd743f334ce796ed4008170aec44 P 9ca6543814fb43abbf580a87de0b53f9: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260808 22:23:32.896369 27825 heartbeater.cc:503] Master 127.9.176.254:38861 was elected leader, sending a full tablet report...
I20260808 22:23:32.896459 28078 tablet_bootstrap.cc:654] T 7551fd743f334ce796ed4008170aec44 P 6cc3bb0ea33149fe96a5170c5e74cc59: Neither blocks nor log segments found. Creating new log.
I20260808 22:23:32.897575 28078 log.cc:824] T 7551fd743f334ce796ed4008170aec44 P 6cc3bb0ea33149fe96a5170c5e74cc59: Log is configured to *not* fsync() on all Append() calls
I20260808 22:23:32.897531 28074 raft_consensus.cc:348] T 7551fd743f334ce796ed4008170aec44 P 6829c2aa81ae4ec9ace4283d33d96159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6829c2aa81ae4ec9ace4283d33d96159" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 45989 } } peers { permanent_uuid: "9ca6543814fb43abbf580a87de0b53f9" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 44251 } } peers { permanent_uuid: "6cc3bb0ea33149fe96a5170c5e74cc59" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40055 } }
I20260808 22:23:32.897782 28074 raft_consensus.cc:374] T 7551fd743f334ce796ed4008170aec44 P 6829c2aa81ae4ec9ace4283d33d96159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:23:32.897840 28074 raft_consensus.cc:729] T 7551fd743f334ce796ed4008170aec44 P 6829c2aa81ae4ec9ace4283d33d96159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6829c2aa81ae4ec9ace4283d33d96159, State: Initialized, Role: FOLLOWER
I20260808 22:23:32.898319 28074 consensus_queue.cc:260] T 7551fd743f334ce796ed4008170aec44 P 6829c2aa81ae4ec9ace4283d33d96159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6829c2aa81ae4ec9ace4283d33d96159" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 45989 } } peers { permanent_uuid: "9ca6543814fb43abbf580a87de0b53f9" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 44251 } } peers { permanent_uuid: "6cc3bb0ea33149fe96a5170c5e74cc59" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40055 } }
I20260808 22:23:32.899439 28074 ts_tablet_manager.cc:1428] T 7551fd743f334ce796ed4008170aec44 P 6829c2aa81ae4ec9ace4283d33d96159: Time spent starting tablet: real 0.007s user 0.004s sys 0.003s
I20260808 22:23:32.899787 27594 heartbeater.cc:503] Master 127.9.176.254:38861 was elected leader, sending a full tablet report...
I20260808 22:23:32.900211 28078 tablet_bootstrap.cc:492] T 7551fd743f334ce796ed4008170aec44 P 6cc3bb0ea33149fe96a5170c5e74cc59: No bootstrap required, opened a new log
I20260808 22:23:32.900431 28078 ts_tablet_manager.cc:1397] T 7551fd743f334ce796ed4008170aec44 P 6cc3bb0ea33149fe96a5170c5e74cc59: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.005s
I20260808 22:23:32.906133 28078 raft_consensus.cc:348] T 7551fd743f334ce796ed4008170aec44 P 6cc3bb0ea33149fe96a5170c5e74cc59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6829c2aa81ae4ec9ace4283d33d96159" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 45989 } } peers { permanent_uuid: "9ca6543814fb43abbf580a87de0b53f9" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 44251 } } peers { permanent_uuid: "6cc3bb0ea33149fe96a5170c5e74cc59" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40055 } }
I20260808 22:23:32.906396 28078 raft_consensus.cc:374] T 7551fd743f334ce796ed4008170aec44 P 6cc3bb0ea33149fe96a5170c5e74cc59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:23:32.906526 28078 raft_consensus.cc:729] T 7551fd743f334ce796ed4008170aec44 P 6cc3bb0ea33149fe96a5170c5e74cc59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cc3bb0ea33149fe96a5170c5e74cc59, State: Initialized, Role: FOLLOWER
I20260808 22:23:32.906970 28078 consensus_queue.cc:260] T 7551fd743f334ce796ed4008170aec44 P 6cc3bb0ea33149fe96a5170c5e74cc59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6829c2aa81ae4ec9ace4283d33d96159" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 45989 } } peers { permanent_uuid: "9ca6543814fb43abbf580a87de0b53f9" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 44251 } } peers { permanent_uuid: "6cc3bb0ea33149fe96a5170c5e74cc59" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40055 } }
I20260808 22:23:32.907996 28078 ts_tablet_manager.cc:1428] T 7551fd743f334ce796ed4008170aec44 P 6cc3bb0ea33149fe96a5170c5e74cc59: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s
I20260808 22:23:32.908356 28046 heartbeater.cc:503] Master 127.9.176.254:38861 was elected leader, sending a full tablet report...
I20260808 22:23:32.964871 28082 raft_consensus.cc:482] T 7551fd743f334ce796ed4008170aec44 P 6829c2aa81ae4ec9ace4283d33d96159 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 22:23:32.965195 28082 raft_consensus.cc:504] T 7551fd743f334ce796ed4008170aec44 P 6829c2aa81ae4ec9ace4283d33d96159 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6829c2aa81ae4ec9ace4283d33d96159" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 45989 } } peers { permanent_uuid: "9ca6543814fb43abbf580a87de0b53f9" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 44251 } } peers { permanent_uuid: "6cc3bb0ea33149fe96a5170c5e74cc59" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40055 } }
I20260808 22:23:32.966754 28082 leader_election.cc:288] T 7551fd743f334ce796ed4008170aec44 P 6829c2aa81ae4ec9ace4283d33d96159 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ca6543814fb43abbf580a87de0b53f9 (127.9.176.194:44251), 6cc3bb0ea33149fe96a5170c5e74cc59 (127.9.176.195:40055)
I20260808 22:23:32.973570 28069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:23:32.968138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.193:33211 (local address 127.9.176.194:44251)
0808 22:23:32.968257 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0808 22:23:32.968261 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:23:32.968648 (+ 387us) server_negotiation.cc:407] Connection header received
0808 22:23:32.968837 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:23:32.968840 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:23:32.968910 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:23:32.968955 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:23:32.969550 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:23:32.970298 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:23:32.972064 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:23:32.972748 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:23:32.972835 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:23:32.972885 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0808 22:23:32.973335 (+ 450us) server_negotiation.cc:299] Negotiation successful
0808 22:23:32.973421 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 22:23:32.974198 28086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:23:32.967668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.194:44251 (local address 127.9.176.193:33211)
0808 22:23:32.968548 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0808 22:23:32.968561 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 22:23:32.968722 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:23:32.969106 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:23:32.969109 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:23:32.969127 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:23:32.969389 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:23:32.969398 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:23:32.970456 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:23:32.970459 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:23:32.971930 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:23:32.971941 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:23:32.973034 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:23:32.973038 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:23:32.973173 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:23:32.973200 (+ 27us) client_negotiation.cc:769] Sending connection context
0808 22:23:32.973952 (+ 752us) client_negotiation.cc:240] Negotiation successful
0808 22:23:32.974038 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":817,"thread_start_us":175,"threads_started":1}
I20260808 22:23:32.975164 27741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7551fd743f334ce796ed4008170aec44" candidate_uuid: "6829c2aa81ae4ec9ace4283d33d96159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ca6543814fb43abbf580a87de0b53f9" is_pre_election: true
I20260808 22:23:32.975512 27741 raft_consensus.cc:2454] T 7551fd743f334ce796ed4008170aec44 P 9ca6543814fb43abbf580a87de0b53f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6829c2aa81ae4ec9ace4283d33d96159 in term 0.
I20260808 22:23:32.976272 27420 leader_election.cc:302] T 7551fd743f334ce796ed4008170aec44 P 6829c2aa81ae4ec9ace4283d33d96159 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6829c2aa81ae4ec9ace4283d33d96159, 9ca6543814fb43abbf580a87de0b53f9; no voters:
I20260808 22:23:32.976588 28082 raft_consensus.cc:2781] T 7551fd743f334ce796ed4008170aec44 P 6829c2aa81ae4ec9ace4283d33d96159 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 22:23:32.976660 28082 raft_consensus.cc:482] T 7551fd743f334ce796ed4008170aec44 P 6829c2aa81ae4ec9ace4283d33d96159 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 22:23:32.976696 28082 raft_consensus.cc:3037] T 7551fd743f334ce796ed4008170aec44 P 6829c2aa81ae4ec9ace4283d33d96159 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:23:32.978790 28082 raft_consensus.cc:504] T 7551fd743f334ce796ed4008170aec44 P 6829c2aa81ae4ec9ace4283d33d96159 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6829c2aa81ae4ec9ace4283d33d96159" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 45989 } } peers { permanent_uuid: "9ca6543814fb43abbf580a87de0b53f9" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 44251 } } peers { permanent_uuid: "6cc3bb0ea33149fe96a5170c5e74cc59" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40055 } }
I20260808 22:23:32.979211 28082 leader_election.cc:288] T 7551fd743f334ce796ed4008170aec44 P 6829c2aa81ae4ec9ace4283d33d96159 [CANDIDATE]: Term 1 election: Requested vote from peers 9ca6543814fb43abbf580a87de0b53f9 (127.9.176.194:44251), 6cc3bb0ea33149fe96a5170c5e74cc59 (127.9.176.195:40055)
I20260808 22:23:32.979755 27741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7551fd743f334ce796ed4008170aec44" candidate_uuid: "6829c2aa81ae4ec9ace4283d33d96159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ca6543814fb43abbf580a87de0b53f9"
I20260808 22:23:32.979882 27741 raft_consensus.cc:3037] T 7551fd743f334ce796ed4008170aec44 P 9ca6543814fb43abbf580a87de0b53f9 [term 0 FOLLOWER]: Advancing to term 1
W20260808 22:23:32.981714 27595 tablet.cc:2369] T 7551fd743f334ce796ed4008170aec44 P 6829c2aa81ae4ec9ace4283d33d96159: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 22:23:32.982422 27741 raft_consensus.cc:2454] T 7551fd743f334ce796ed4008170aec44 P 9ca6543814fb43abbf580a87de0b53f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6829c2aa81ae4ec9ace4283d33d96159 in term 1.
I20260808 22:23:32.982764 27420 leader_election.cc:302] T 7551fd743f334ce796ed4008170aec44 P 6829c2aa81ae4ec9ace4283d33d96159 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6829c2aa81ae4ec9ace4283d33d96159, 9ca6543814fb43abbf580a87de0b53f9; no voters:
I20260808 22:23:32.982976 28082 raft_consensus.cc:2781] T 7551fd743f334ce796ed4008170aec44 P 6829c2aa81ae4ec9ace4283d33d96159 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:23:32.983167 28082 raft_consensus.cc:686] T 7551fd743f334ce796ed4008170aec44 P 6829c2aa81ae4ec9ace4283d33d96159 [term 1 LEADER]: Becoming Leader. State: Replica: 6829c2aa81ae4ec9ace4283d33d96159, State: Running, Role: LEADER
I20260808 22:23:32.983493 28082 consensus_queue.cc:237] T 7551fd743f334ce796ed4008170aec44 P 6829c2aa81ae4ec9ace4283d33d96159 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6829c2aa81ae4ec9ace4283d33d96159" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 45989 } } peers { permanent_uuid: "9ca6543814fb43abbf580a87de0b53f9" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 44251 } } peers { permanent_uuid: "6cc3bb0ea33149fe96a5170c5e74cc59" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40055 } }
I20260808 22:23:32.986575 28072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:23:32.967287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.193:60253 (local address 127.9.176.195:40055)
0808 22:23:32.967444 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0808 22:23:32.967449 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:23:32.968297 (+ 848us) server_negotiation.cc:407] Connection header received
0808 22:23:32.982211 (+ 13914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:23:32.982215 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:23:32.982273 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:23:32.982327 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:23:32.982853 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:23:32.983601 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:23:32.985270 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:23:32.985827 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:23:32.985922 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:23:32.985983 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0808 22:23:32.986379 (+ 396us) server_negotiation.cc:299] Negotiation successful
0808 22:23:32.986457 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 22:23:32.986744 27257 catalog_manager.cc:5495] T 7551fd743f334ce796ed4008170aec44 P 6829c2aa81ae4ec9ace4283d33d96159 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6829c2aa81ae4ec9ace4283d33d96159 (127.9.176.193). New cstate: current_term: 1 leader_uuid: "6829c2aa81ae4ec9ace4283d33d96159" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6829c2aa81ae4ec9ace4283d33d96159" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 45989 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ca6543814fb43abbf580a87de0b53f9" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 44251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6cc3bb0ea33149fe96a5170c5e74cc59" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40055 } health_report { overall_health: UNKNOWN } } }
I20260808 22:23:32.987277 28084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:23:32.967212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.195:40055 (local address 127.9.176.193:60253)
0808 22:23:32.968156 (+ 944us) negotiation.cc:107] Waiting for socket to connect
0808 22:23:32.968172 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 22:23:32.982018 (+ 13846us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:23:32.982487 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:23:32.982490 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:23:32.982508 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:23:32.982715 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:23:32.982722 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:23:32.983844 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:23:32.983847 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:23:32.985118 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:23:32.985130 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:23:32.986075 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:23:32.986079 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:23:32.986225 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:23:32.986253 (+ 28us) client_negotiation.cc:769] Sending connection context
0808 22:23:32.986762 (+ 509us) client_negotiation.cc:240] Negotiation successful
0808 22:23:32.986838 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":862,"thread_start_us":122,"threads_started":1}
W20260808 22:23:32.991044 28047 tablet.cc:2369] T 7551fd743f334ce796ed4008170aec44 P 6cc3bb0ea33149fe96a5170c5e74cc59: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 22:23:32.992624 27974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7551fd743f334ce796ed4008170aec44" candidate_uuid: "6829c2aa81ae4ec9ace4283d33d96159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc3bb0ea33149fe96a5170c5e74cc59" is_pre_election: true
I20260808 22:23:32.992967 27974 raft_consensus.cc:2454] T 7551fd743f334ce796ed4008170aec44 P 6cc3bb0ea33149fe96a5170c5e74cc59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6829c2aa81ae4ec9ace4283d33d96159 in term 0.
I20260808 22:23:32.993036 27972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7551fd743f334ce796ed4008170aec44" candidate_uuid: "6829c2aa81ae4ec9ace4283d33d96159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc3bb0ea33149fe96a5170c5e74cc59"
I20260808 22:23:32.993162 27972 raft_consensus.cc:3037] T 7551fd743f334ce796ed4008170aec44 P 6cc3bb0ea33149fe96a5170c5e74cc59 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:23:32.994807 27972 raft_consensus.cc:2454] T 7551fd743f334ce796ed4008170aec44 P 6cc3bb0ea33149fe96a5170c5e74cc59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6829c2aa81ae4ec9ace4283d33d96159 in term 1.
I20260808 22:23:33.005391 28067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:23:33.000971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53708 (local address 127.9.176.193:45989)
0808 22:23:33.001088 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0808 22:23:33.001092 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:23:33.001107 (+ 15us) server_negotiation.cc:407] Connection header received
0808 22:23:33.001244 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:23:33.001248 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:23:33.001308 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:23:33.001417 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:23:33.001681 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:23:33.002454 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:23:33.003691 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:23:33.004030 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:23:33.004141 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:23:33.004433 (+ 292us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:23:33.004993 (+ 560us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:23:33.005118 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0808 22:23:33.005200 (+ 82us) server_negotiation.cc:299] Negotiation successful
0808 22:23:33.005263 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 22:23:33.011845 27972 raft_consensus.cc:1261] T 7551fd743f334ce796ed4008170aec44 P 6cc3bb0ea33149fe96a5170c5e74cc59 [term 1 FOLLOWER]: Refusing update from remote peer 6829c2aa81ae4ec9ace4283d33d96159: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:23:33.011849 27741 raft_consensus.cc:1261] T 7551fd743f334ce796ed4008170aec44 P 9ca6543814fb43abbf580a87de0b53f9 [term 1 FOLLOWER]: Refusing update from remote peer 6829c2aa81ae4ec9ace4283d33d96159: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:23:33.012550 28082 consensus_queue.cc:1035] T 7551fd743f334ce796ed4008170aec44 P 6829c2aa81ae4ec9ace4283d33d96159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ca6543814fb43abbf580a87de0b53f9" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 44251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:23:33.013425 28087 consensus_queue.cc:1035] T 7551fd743f334ce796ed4008170aec44 P 6829c2aa81ae4ec9ace4283d33d96159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc3bb0ea33149fe96a5170c5e74cc59" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 40055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:23:33.023835 28091 mvcc.cc:205] Tried to move back new op lower bound from 7316389122089734144 to 7316389121983885312. Current Snapshot: MvccSnapshot[applied={T|T < 7316389122089734144}]
I20260808 22:23:33.026638 28092 mvcc.cc:205] Tried to move back new op lower bound from 7316389122089734144 to 7316389121983885312. Current Snapshot: MvccSnapshot[applied={T|T < 7316389122089734144}]
I20260808 22:23:33.031587 28093 mvcc.cc:205] Tried to move back new op lower bound from 7316389122089734144 to 7316389121983885312. Current Snapshot: MvccSnapshot[applied={T|T < 7316389122089734144}]
I20260808 22:23:33.038524 27257 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40634:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:23:33.038705 27257 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40634:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:23:33.041628 27257 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7316be129c714498aa24cb018a9c2718: Sending DeleteTablet for 3 replicas of tablet 7551fd743f334ce796ed4008170aec44
I20260808 22:23:33.042698 27493 tablet_service.cc:1514] Processing DeleteTablet for tablet 7551fd743f334ce796ed4008170aec44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:23:33 UTC) from {username='kudu', principal='kudu/127.9.176.254@KRBTEST.COM'} at 127.0.0.1:53704
I20260808 22:23:33.043705 27705 tablet_service.cc:1514] Processing DeleteTablet for tablet 7551fd743f334ce796ed4008170aec44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:23:33 UTC) from {username='kudu', principal='kudu/127.9.176.254@KRBTEST.COM'} at 127.0.0.1:33948
I20260808 22:23:33.044730 27932 tablet_service.cc:1514] Processing DeleteTablet for tablet 7551fd743f334ce796ed4008170aec44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:23:33 UTC) from {username='kudu', principal='kudu/127.9.176.254@KRBTEST.COM'} at 127.0.0.1:53962
I20260808 22:23:33.045913 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 27392
I20260808 22:23:33.049585 28103 tablet_replica.cc:330] T 7551fd743f334ce796ed4008170aec44 P 9ca6543814fb43abbf580a87de0b53f9: stopping tablet replica
I20260808 22:23:33.049949 28103 raft_consensus.cc:2229] T 7551fd743f334ce796ed4008170aec44 P 9ca6543814fb43abbf580a87de0b53f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:23:33.050345 28105 tablet_replica.cc:330] T 7551fd743f334ce796ed4008170aec44 P 6cc3bb0ea33149fe96a5170c5e74cc59: stopping tablet replica
I20260808 22:23:33.051858 28103 raft_consensus.cc:2258] T 7551fd743f334ce796ed4008170aec44 P 9ca6543814fb43abbf580a87de0b53f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:23:33.052238 28105 raft_consensus.cc:2229] T 7551fd743f334ce796ed4008170aec44 P 6cc3bb0ea33149fe96a5170c5e74cc59 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:23:33.052917 28105 raft_consensus.cc:2258] T 7551fd743f334ce796ed4008170aec44 P 6cc3bb0ea33149fe96a5170c5e74cc59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:23:33.054690 28103 ts_tablet_manager.cc:1905] T 7551fd743f334ce796ed4008170aec44 P 9ca6543814fb43abbf580a87de0b53f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:23:33.056596 28105 ts_tablet_manager.cc:1905] T 7551fd743f334ce796ed4008170aec44 P 6cc3bb0ea33149fe96a5170c5e74cc59: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:23:33.059793 28103 ts_tablet_manager.cc:1918] T 7551fd743f334ce796ed4008170aec44 P 9ca6543814fb43abbf580a87de0b53f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 22:23:33.062433 28105 ts_tablet_manager.cc:1918] T 7551fd743f334ce796ed4008170aec44 P 6cc3bb0ea33149fe96a5170c5e74cc59: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 22:23:33.062577 28105 log.cc:1196] T 7551fd743f334ce796ed4008170aec44 P 6cc3bb0ea33149fe96a5170c5e74cc59: Deleting WAL directory at /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786227706282457-9923-0/minicluster-data/ts-2/wal/wals/7551fd743f334ce796ed4008170aec44
W20260808 22:23:33.063208 27232 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.176.193:45989 (error 108)
I20260808 22:23:33.063266 28105 ts_tablet_manager.cc:1939] T 7551fd743f334ce796ed4008170aec44 P 6cc3bb0ea33149fe96a5170c5e74cc59: Deleting consensus metadata
I20260808 22:23:33.064369 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 27610
I20260808 22:23:33.069656 27234 catalog_manager.cc:4841] TS 6cc3bb0ea33149fe96a5170c5e74cc59 (127.9.176.195:40055): tablet 7551fd743f334ce796ed4008170aec44 (table test-table [id=f7043367cab847aba2238fdbb68d8122]) successfully deleted
I20260808 22:23:33.072386 28068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:23:33.068642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.193:45989 (local address 127.0.0.1:53716)
0808 22:23:33.068860 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0808 22:23:33.072291 (+ 3431us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.193:45989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
W20260808 22:23:33.072676 27232 catalog_manager.cc:4584] TS 6829c2aa81ae4ec9ace4283d33d96159 (127.9.176.193:45989): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7551fd743f334ce796ed4008170aec44: Network error: Client connection negotiation failed: client connection to 127.9.176.193:45989: connect: Connection refused (error 111)
I20260808 22:23:33.078811 28068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:23:33.078577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.194:44251 (local address 127.0.0.1:33962)
0808 22:23:33.078687 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0808 22:23:33.078737 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.194:44251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260808 22:23:33.079082 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 27833
I20260808 22:23:33.089717 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 27201
I20260808 22:23:33.101657 9923 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260808 22:23:33.445856 9923 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260808 22:23:33.446017 9923 mini_ranger.cc:66] Stopping Ranger...
I20260808 22:23:33.789996 9923 mini_ranger.cc:68] Stopped Ranger
2026-08-08 22:23:33.790 UTC [26567] LOG: received smart shutdown request
2026-08-08 22:23:33.794 UTC [26567] LOG: background worker "logical replication launcher" (PID 26581) exited with exit code 1
2026-08-08 22:23:33.794 UTC [26574] LOG: shutting down
2026-08-08 22:23:33.948 UTC [26567] LOG: database system is shut down
2026-08-08T22:23:33Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (45414 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 22:23:34 dist-test-slave-ncp9 krb5kdc[28123](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 22:23:34 dist-test-slave-ncp9 krb5kdc[28123](info): set up 2 sockets
Aug 08 22:23:34 dist-test-slave-ncp9 krb5kdc[28123](info): commencing operation
krb5kdc: starting...
W20260808 22:23:35.194506 9923 mini_kdc.cc:121] Time spent starting KDC: real 1.131s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/krb5kdc/test-user.keytab.
Aug 08 22:23:35 dist-test-slave-ncp9 krb5kdc[28123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T22:23:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:23:35Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.9.176.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.9.176.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.9.176.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/krb5kdc/rangeradmin_127.9.176.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.9.176.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/krb5kdc/rangeradmin_127.9.176.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.9.176.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.9.176.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.9.176.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/krb5kdc/rangerlookup_127.9.176.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.9.176.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/krb5kdc/rangerlookup_127.9.176.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.9.176.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.176.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.176.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/krb5kdc/HTTP_127.9.176.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.9.176.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/krb5kdc/HTTP_127.9.176.212@KRBTEST.COM.keytab.
I20260808 22:23:35.362016 9923 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskqqDyFd/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/postgres -l logfile start
2026-08-08 22:23:37.764 UTC [28163] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-08 22:23:37.764 UTC [28163] LOG: listening on IPv4 address "127.9.176.212", port 58867
2026-08-08 22:23:37.767 UTC [28163] LOG: listening on Unix socket "/tmp/.s.PGSQL.58867"
2026-08-08 22:23:37.787 UTC [28168] LOG: database system was shut down at 2026-08-08 22:23:36 UTC
2026-08-08 22:23:37.790 UTC [28163] LOG: database system is ready to accept connections
I20260808 22:23:38.831059 9923 mini_postgres.cc:96] Postgres bound to 58867
2026-08-08 22:23:38.838 UTC [28181] FATAL: database "slave" does not exist
127.9.176.212:58867 - accepting connections
I20260808 22:23:38.838550 9923 mini_ranger.cc:161] Starting Ranger...
I20260808 22:23:38.853317 9923 mini_ranger.cc:84] Created miniranger Postgres user
I20260808 22:23:39.314885 9923 mini_ranger.cc:87] Created ranger Postgres database
I20260808 22:23:39.314970 9923 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ranger-admin
2026-08-08 22:23:39,621 [I] DB FLAVOR :POSTGRES
2026-08-08 22:23:39,622 [I] --------- Verifying Ranger DB connection ---------
2026-08-08 22:23:39,622 [I] Checking connection..
2026-08-08 22:23:39,622 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqqDyFd/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqqDyFd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.176.212:58867/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-08 22:23:39,873 [I] Checking connection passed.
2026-08-08 22:23:39,873 [I] --------- Verifying version history table ---------
2026-08-08 22:23:39,873 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqqDyFd/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqqDyFd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.176.212:58867/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 22:23:40,138 [I] Table x_db_version_h does not exist in database ranger
2026-08-08 22:23:40,139 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqqDyFd/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqqDyFd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.176.212:58867/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 22:23:40,382 [I] Table x_db_version_h does not exist in database ranger
2026-08-08 22:23:40,383 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-08 22:23:40,383 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqqDyFd/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqqDyFd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.176.212:58867/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskqqDyFd/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-08 22:23:40.610 UTC [28274] WARNING: there is no transaction in progress
2026-08-08 22:23:40,618 [I] create_dbversion_catalog.sql file imported successfully
2026-08-08 22:23:40,618 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqqDyFd/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqqDyFd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.176.212:58867/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 22:23:40,882 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-08 22:23:40,882 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-08 22:23:40,882 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqqDyFd/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqqDyFd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.176.212:58867/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-08 22:23:41,122 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqqDyFd/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqqDyFd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.176.212:58867/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-08 22:23:41,375 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqqDyFd/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqqDyFd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.176.212:58867/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-ncp9.c.gcp-upstream.internal','N') ;"
2026-08-08 22:23:41,608 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqqDyFd/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqqDyFd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.176.212:58867/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-08 22:23:41,868 [I] Table x_portal_user does not exist in database ranger
2026-08-08 22:23:41,868 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqqDyFd/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqqDyFd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.176.212:58867/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-08 22:23:42,128 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-08 22:23:42,128 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqqDyFd/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqqDyFd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.176.212:58867/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-08 22:23:42,385 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqqDyFd/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqqDyFd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.176.212:58867/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-08 22:23:42,622 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-08 22:23:42,622 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqqDyFd/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqqDyFd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.176.212:58867/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskqqDyFd/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-08 22:23:44.074 UTC [28433] WARNING: there is no transaction in progress
2026-08-08 22:23:44.095 UTC [28433] WARNING: there is no transaction in progress
2026-08-08 22:23:44.119 UTC [28433] WARNING: there is no transaction in progress
2026-08-08 22:23:44.140 UTC [28433] WARNING: there is no transaction in progress
2026-08-08 22:23:44.163 UTC [28433] WARNING: there is no transaction in progress
2026-08-08 22:23:44.187 UTC [28433] WARNING: there is no transaction in progress
2026-08-08 22:23:44.268 UTC [28433] WARNING: there is no transaction in progress
2026-08-08 22:23:44.278 UTC [28433] WARNING: there is no transaction in progress
2026-08-08 22:23:44.294 UTC [28433] WARNING: there is no transaction in progress
2026-08-08 22:23:44.304 UTC [28433] WARNING: there is no transaction in progress
2026-08-08 22:23:44.314 UTC [28433] WARNING: there is no transaction in progress
2026-08-08 22:23:44.327 UTC [28433] WARNING: there is no transaction in progress
2026-08-08 22:23:44.336 UTC [28433] WARNING: there is no transaction in progress
2026-08-08 22:23:44.342 UTC [28433] WARNING: there is no transaction in progress
2026-08-08 22:23:44.348 UTC [28433] WARNING: there is no transaction in progress
2026-08-08 22:23:44.355 UTC [28433] WARNING: there is no transaction in progress
2026-08-08 22:23:44,936 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-08 22:23:44,936 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqqDyFd/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqqDyFd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.176.212:58867/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-08 22:23:45,175 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-08 22:23:45,175 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqqDyFd/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqqDyFd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.176.212:58867/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-08 22:23:45,422 [I] Table x_portal_user already exists in database 'ranger'
2026-08-08 22:23:45,423 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqqDyFd/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqqDyFd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.176.212:58867/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-08 22:23:45,671 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-08 22:23:45,671 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqqDyFd/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqqDyFd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.176.212:58867/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-08 22:23:45,912 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqqDyFd/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqqDyFd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.176.212:58867/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-08 22:23:46,161 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqqDyFd/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqqDyFd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.176.212:58867/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-ncp9.c.gcp-upstream.internal';"
2026-08-08 22:23:46,404 [I] CORE_DB_SCHEMA import status has been updated
2026-08-08 22:23:46,405 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqqDyFd/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqqDyFd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.176.212:58867/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-08 22:23:46,652 [I] DB_PATCHES have already been applied
I20260808 22:23:46.657856 9923 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ranger-admin:/tmp/dist-test-taskqqDyFd/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskqqDyFd/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskqqDyFd/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqqDyFd/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260808 22:23:46.657925 9923 mini_ranger.cc:193] Using host: 127.9.176.212
I20260808 22:23:46.662035 9923 mini_ranger.cc:227] Ranger admin URL: http://127.9.176.212:41741
Aug 08, 2026 10:23:46 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 08, 2026 10:23:46 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 08, 2026 10:23:47 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ranger-admin/ews/webapp
Aug 08, 2026 10:23:47 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 08, 2026 10:23:47 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 08, 2026 10:23:47 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ranger-admin/ews/webapp].
Aug 08, 2026 10:23:47 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260808 22:23:47.889736 9923 mini_ranger.cc:160] Time spent starting Ranger: real 9.051s user 0.005s sys 0.005s
Aug 08, 2026 10:24:06 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 08, 2026 10:24:06 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 08, 2026 10:24:06 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 08, 2026 10:24:06 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 08, 2026 10:24:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 08, 2026 10:24:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 08, 2026 10:24:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 08, 2026 10:24:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 08, 2026 10:24:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 08, 2026 10:24:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 08, 2026 10:24:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 08, 2026 10:24:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 08, 2026 10:24:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 08, 2026 10:24:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 08, 2026 10:24:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 08, 2026 10:24:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 08, 2026 10:24:06 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 08, 2026 10:24:07 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260808 22:24:08.189175 9923 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.9.176.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.9.176.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.9.176.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/krb5kdc/rangerkms_127.9.176.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.9.176.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/krb5kdc/rangerkms_127.9.176.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.9.176.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.9.176.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.176.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/krb5kdc/HTTP_127.9.176.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.9.176.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/krb5kdc/HTTP_127.9.176.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 08 22:24:08 dist-test-slave-ncp9 krb5kdc[28123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227848, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260808 22:24:08.287043 9923 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260808 22:24:08.297330 9923 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260808 22:24:09.019510 9923 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260808 22:24:09.019596 9923 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ranger-kms
I20260808 22:24:09.019611 9923 mini_ranger_kms.cc:227] Using postgres at 127.9.176.212:58867
2026-08-08 22:24:09,180 [I] DB FLAVOR :POSTGRES
2026-08-08 22:24:09,180 [I] --------- Verifying Ranger DB connection ---------
2026-08-08 22:24:09,180 [I] Checking connection
2026-08-08 22:24:09,180 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqqDyFd/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.176.212:58867/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-08 22:24:09,413 [I] connection success
2026-08-08 22:24:09,414 [I] --------- Verifying Ranger DB tables ---------
2026-08-08 22:24:09,414 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-08 22:24:09,414 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqqDyFd/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.176.212:58867/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-08 22:24:09,650 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-08 22:24:09,650 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-08 22:24:09,650 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-08 22:24:09,651 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqqDyFd/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.176.212:58867/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-08 22:24:09,893 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260808 22:24:10.312120 9923 mini_ranger_kms.cc:310] Created kms service
I20260808 22:24:10.477372 9923 mini_ranger_kms.cc:326] Created kudu user
I20260808 22:24:10.520797 9923 mini_ranger_kms.cc:343] Created rangerkms user
Aug 08 22:24:10 dist-test-slave-ncp9 krb5kdc[28123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227848, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.9.176.212@KRBTEST.COM
I20260808 22:24:10.897246 9923 mini_ranger_kms.cc:384] Added ranger policy
I20260808 22:24:10.897403 9923 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ranger-kms:/tmp/dist-test-taskqqDyFd/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqqDyFd/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskqqDyFd/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260808 22:24:10.897423 9923 mini_ranger_kms.cc:240] Using host: 127.9.176.212
I20260808 22:24:10.902007 9923 mini_ranger_kms.cc:276] Ranger KMS PID: 28705
I20260808 22:24:10.902093 9923 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.9.176.212:35113
Aug 08, 2026 10:24:11 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 08, 2026 10:24:11 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 08, 2026 10:24:11 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 08, 2026 10:24:11 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 08, 2026 10:24:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ranger-kms/ews/webapp].
Aug 08, 2026 10:24:12 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 08 22:24:12 dist-test-slave-ncp9 krb5kdc[28123](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786227852, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.176.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 08 22:24:13 dist-test-slave-ncp9 krb5kdc[28123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227852, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.176.212@KRBTEST.COM for HTTP/127.9.176.212@KRBTEST.COM
I20260808 22:24:13.361186 9923 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.074s user 0.018s sys 0.127s
I20260808 22:24:13.361310 9923 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260808 22:24:13.361372 9923 mini_ranger_kms.cc:401] 127.9.176.212:35113/kms/v1/keys
Aug 08 22:24:13 dist-test-slave-ncp9 krb5kdc[28123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227852, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.176.212@KRBTEST.COM for HTTP/127.9.176.212@KRBTEST.COM
Aug 08 22:24:15 dist-test-slave-ncp9 krb5kdc[28123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.9.176.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.176.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.176.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab.
Entry for principal kudu/127.9.176.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab.
I20260808 22:24:15.963608 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:42379
--webserver_interface=127.9.176.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab
--principal=kudu/127.9.176.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:43091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.176.254:42379
--ranger_config_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.176.212:35113/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:24:16.075241 28783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:24:16.075495 28783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:24:16.075547 28783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:24:16.079435 28783 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:24:16.079488 28783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:24:16.079514 28783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:24:16.079535 28783 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:24:16.079560 28783 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:24:16.084749 28783 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:43091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/master-0/wal
--ranger_kms_url=127.9.176.212:35113/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.9.176.254:42379
--ranger_config_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab
--principal=kudu/127.9.176.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:42379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.176.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28783
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:24:16.085731 28783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:24:16.086704 28783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:24:16.095342 28794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:24:16.095348 28789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:24:16.095386 28783 server_base.cc:1034] running on GCE node
W20260808 22:24:16.095350 28790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:24:16.096124 28783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:24:16.097249 28783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:24:16.098407 28783 hybrid_clock.cc:648] HybridClock initialized: now 1786227856098387 us; error 33 us; skew 500 ppm
Aug 08 22:24:16 dist-test-slave-ncp9 krb5kdc[28123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227856, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:24:16.101084 28783 init.cc:377] Logged in from keytab as kudu/127.9.176.254@KRBTEST.COM (short username kudu)
I20260808 22:24:16.101895 28783 webserver.cc:460] Webserver started at http://127.9.176.254:44953/ using document root <none> and password file <none>
I20260808 22:24:16.102398 28783 fs_manager.cc:359] Metadata directory not provided
I20260808 22:24:16.102450 28783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:24:16.102609 28783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 22:24:16 dist-test-slave-ncp9 krb5kdc[28123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227856, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.254@KRBTEST.COM for HTTP/127.9.176.212@KRBTEST.COM
I20260808 22:24:16.149041 28783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/master-0/data/instance:
uuid: "ece099de893c46439733749788a8d611"
format_stamp: "Formatted at 2026-08-08 22:24:16 on dist-test-slave-ncp9"
server_key: "25516f32aae133bf91ed449ea49cbade"
server_key_iv: "75590ebbbdf2a6f8cfc27b4307ee0688"
server_key_version: "kuduclusterkey@0"
I20260808 22:24:16.149655 28783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/master-0/wal/instance:
uuid: "ece099de893c46439733749788a8d611"
format_stamp: "Formatted at 2026-08-08 22:24:16 on dist-test-slave-ncp9"
server_key: "25516f32aae133bf91ed449ea49cbade"
server_key_iv: "75590ebbbdf2a6f8cfc27b4307ee0688"
server_key_version: "kuduclusterkey@0"
I20260808 22:24:16.153632 28783 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260808 22:24:16.169747 28802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:24:16.171447 28783 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 22:24:16.171592 28783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/master-0/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/master-0/wal
uuid: "ece099de893c46439733749788a8d611"
format_stamp: "Formatted at 2026-08-08 22:24:16 on dist-test-slave-ncp9"
server_key: "25516f32aae133bf91ed449ea49cbade"
server_key_iv: "75590ebbbdf2a6f8cfc27b4307ee0688"
server_key_version: "kuduclusterkey@0"
I20260808 22:24:16.171736 28783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:24:16.208976 28783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:24:16.212422 28783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:24:16.212641 28783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:24:16.221462 28783 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.254:42379
I20260808 22:24:16.221470 28871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.254:42379 every 8 connection(s)
I20260808 22:24:16.222688 28783 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
I20260808 22:24:16.225698 28872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:24:16.231182 28872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ece099de893c46439733749788a8d611: Bootstrap starting.
I20260808 22:24:16.231294 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 28783
I20260808 22:24:16.231415 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/master-0/wal/instance
I20260808 22:24:16.231688 9923 external_mini_cluster.cc:1428] Setting key 0f7b451880cb1995bbc76eb48eb690f4
I20260808 22:24:16.233685 28872 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ece099de893c46439733749788a8d611: Neither blocks nor log segments found. Creating new log.
I20260808 22:24:16.234380 28872 log.cc:824] T 00000000000000000000000000000000 P ece099de893c46439733749788a8d611: Log is configured to *not* fsync() on all Append() calls
I20260808 22:24:16.236364 28872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ece099de893c46439733749788a8d611: No bootstrap required, opened a new log
Aug 08 22:24:16 dist-test-slave-ncp9 krb5kdc[28123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
I20260808 22:24:16.240365 28872 raft_consensus.cc:348] T 00000000000000000000000000000000 P ece099de893c46439733749788a8d611 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ece099de893c46439733749788a8d611" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 42379 } }
I20260808 22:24:16.240657 28872 raft_consensus.cc:374] T 00000000000000000000000000000000 P ece099de893c46439733749788a8d611 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:24:16.240721 28872 raft_consensus.cc:729] T 00000000000000000000000000000000 P ece099de893c46439733749788a8d611 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ece099de893c46439733749788a8d611, State: Initialized, Role: FOLLOWER
I20260808 22:24:16.241195 28872 consensus_queue.cc:260] T 00000000000000000000000000000000 P ece099de893c46439733749788a8d611 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ece099de893c46439733749788a8d611" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 42379 } }
I20260808 22:24:16.241323 28872 raft_consensus.cc:388] T 00000000000000000000000000000000 P ece099de893c46439733749788a8d611 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:24:16.241390 28872 raft_consensus.cc:482] T 00000000000000000000000000000000 P ece099de893c46439733749788a8d611 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:24:16.241487 28872 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ece099de893c46439733749788a8d611 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:24:16.242442 28872 raft_consensus.cc:504] T 00000000000000000000000000000000 P ece099de893c46439733749788a8d611 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ece099de893c46439733749788a8d611" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 42379 } }
I20260808 22:24:16.242779 28872 leader_election.cc:302] T 00000000000000000000000000000000 P ece099de893c46439733749788a8d611 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ece099de893c46439733749788a8d611; no voters:
I20260808 22:24:16.243139 28872 leader_election.cc:288] T 00000000000000000000000000000000 P ece099de893c46439733749788a8d611 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:24:16.243224 28877 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ece099de893c46439733749788a8d611 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:24:16.243443 28877 raft_consensus.cc:686] T 00000000000000000000000000000000 P ece099de893c46439733749788a8d611 [term 1 LEADER]: Becoming Leader. State: Replica: ece099de893c46439733749788a8d611, State: Running, Role: LEADER
I20260808 22:24:16.243821 28877 consensus_queue.cc:237] T 00000000000000000000000000000000 P ece099de893c46439733749788a8d611 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ece099de893c46439733749788a8d611" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 42379 } }
I20260808 22:24:16.244350 28872 sys_catalog.cc:564] T 00000000000000000000000000000000 P ece099de893c46439733749788a8d611 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:24:16.245476 28879 sys_catalog.cc:455] T 00000000000000000000000000000000 P ece099de893c46439733749788a8d611 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ece099de893c46439733749788a8d611" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ece099de893c46439733749788a8d611" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 42379 } } }
I20260808 22:24:16.245597 28879 sys_catalog.cc:458] T 00000000000000000000000000000000 P ece099de893c46439733749788a8d611 [sys.catalog]: This master's current role is: LEADER
I20260808 22:24:16.245800 28879 sys_catalog.cc:455] T 00000000000000000000000000000000 P ece099de893c46439733749788a8d611 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ece099de893c46439733749788a8d611. Latest consensus state: current_term: 1 leader_uuid: "ece099de893c46439733749788a8d611" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ece099de893c46439733749788a8d611" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 42379 } } }
I20260808 22:24:16.245862 28879 sys_catalog.cc:458] T 00000000000000000000000000000000 P ece099de893c46439733749788a8d611 [sys.catalog]: This master's current role is: LEADER
I20260808 22:24:16.246430 28872 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260808 22:24:16.246572 28875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:16.233014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39256 (local address 127.9.176.254:42379)
0808 22:24:16.233417 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0808 22:24:16.233434 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 22:24:16.233469 (+ 35us) server_negotiation.cc:407] Connection header received
0808 22:24:16.234112 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:24:16.234129 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:24:16.234452 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:24:16.234779 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:24:16.235550 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:16.236563 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:16.237735 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:16.238196 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:16.238343 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:16.241039 (+ 2696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:24:16.241045 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:24:16.241062 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:24:16.241093 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:24:16.242724 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:24:16.243229 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:24:16.243236 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:24:16.243242 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:24:16.243380 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:24:16.243725 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:24:16.243728 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:24:16.243730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:24:16.244231 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:24:16.244407 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0808 22:24:16.245289 (+ 882us) server_negotiation.cc:299] Negotiation successful
0808 22:24:16.245592 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":112,"threads_started":1}
I20260808 22:24:16.246714 28893 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:24:16.252990 28893 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:24:16.259438 28893 catalog_manager.cc:1347] Generated new cluster ID: 99eb7227a6e64a5a8a574dec6b0afe3d
I20260808 22:24:16.259588 28893 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:24:16.266822 28893 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:24:16.268087 28893 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:24:16.273253 28893 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ece099de893c46439733749788a8d611: Generated new TSK 0
I20260808 22:24:16.273994 28893 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.176.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.176.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.176.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.193.keytab.
Entry for principal kudu/127.9.176.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.193.keytab.
I20260808 22:24:16.916887 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.176.212:35113/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.193:0
--local_ip_for_outbound_sockets=127.9.176.193
--webserver_interface=127.9.176.193
--webserver_port=0
--tserver_master_addrs=127.9.176.254:42379
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.193.keytab
--principal=kudu/127.9.176.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:43091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:24:17.058552 28936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:24:17.058872 28936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:24:17.058950 28936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:24:17.064792 28936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:24:17.064864 28936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:24:17.064963 28936 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.193
I20260808 22:24:17.072131 28936 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:43091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.9.176.212:35113/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.193.keytab
--principal=kudu/127.9.176.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.176.193
--webserver_port=0
--tserver_master_addrs=127.9.176.254:42379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28936
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.193
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:24:17.073699 28936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:24:17.075083 28936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:24:17.086457 28948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:24:17.086453 28946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:24:17.086711 28936 server_base.cc:1034] running on GCE node
W20260808 22:24:17.086643 28945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:24:17.087563 28936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:24:17.088291 28936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:24:17.089509 28936 hybrid_clock.cc:648] HybridClock initialized: now 1786227857089483 us; error 52 us; skew 500 ppm
Aug 08 22:24:17 dist-test-slave-ncp9 krb5kdc[28123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227857, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:24:17.093036 28936 init.cc:377] Logged in from keytab as kudu/127.9.176.193@KRBTEST.COM (short username kudu)
I20260808 22:24:17.094095 28936 webserver.cc:460] Webserver started at http://127.9.176.193:42917/ using document root <none> and password file <none>
I20260808 22:24:17.094842 28936 fs_manager.cc:359] Metadata directory not provided
I20260808 22:24:17.094938 28936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:24:17.095198 28936 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 22:24:17 dist-test-slave-ncp9 krb5kdc[28123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227857, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.193@KRBTEST.COM for HTTP/127.9.176.212@KRBTEST.COM
I20260808 22:24:17.110526 28936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ts-0/data/instance:
uuid: "9a25027701d348159e49952160a3e2f2"
format_stamp: "Formatted at 2026-08-08 22:24:17 on dist-test-slave-ncp9"
server_key: "04dc315f4cb42833148e6ea486a2d39e"
server_key_iv: "b1e68456936f37514602129ec02440ff"
server_key_version: "kuduclusterkey@0"
I20260808 22:24:17.111353 28936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ts-0/wal/instance:
uuid: "9a25027701d348159e49952160a3e2f2"
format_stamp: "Formatted at 2026-08-08 22:24:17 on dist-test-slave-ncp9"
server_key: "04dc315f4cb42833148e6ea486a2d39e"
server_key_iv: "b1e68456936f37514602129ec02440ff"
server_key_version: "kuduclusterkey@0"
I20260808 22:24:17.118649 28936 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.000s sys 0.004s
I20260808 22:24:17.137281 28958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:24:17.139024 28936 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.000s
I20260808 22:24:17.139257 28936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ts-0/wal
uuid: "9a25027701d348159e49952160a3e2f2"
format_stamp: "Formatted at 2026-08-08 22:24:17 on dist-test-slave-ncp9"
server_key: "04dc315f4cb42833148e6ea486a2d39e"
server_key_iv: "b1e68456936f37514602129ec02440ff"
server_key_version: "kuduclusterkey@0"
I20260808 22:24:17.139495 28936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:24:17.183281 28936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:24:17.187099 28936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:24:17.187347 28936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:24:17.188117 28936 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:24:17.189093 28936 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:24:17.189204 28936 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:24:17.189334 28936 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:24:17.189366 28936 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:24:17.199216 28936 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.193:36751
I20260808 22:24:17.199257 29123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.193:36751 every 8 connection(s)
I20260808 22:24:17.200297 28936 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
I20260808 22:24:17.209746 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 28936
I20260808 22:24:17.209861 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ts-0/wal/instance
I20260808 22:24:17.210170 9923 external_mini_cluster.cc:1428] Setting key 2ef61b75669e02193ea4448eac88f9b4
Aug 08 22:24:17 dist-test-slave-ncp9 krb5kdc[28123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227857, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.193@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
I20260808 22:24:17.222608 29127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:17.202345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.193:46195 (local address 127.9.176.254:42379)
0808 22:24:17.203773 (+ 1428us) server_negotiation.cc:206] Beginning negotiation
0808 22:24:17.203784 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 22:24:17.203801 (+ 17us) server_negotiation.cc:407] Connection header received
0808 22:24:17.204475 (+ 674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:24:17.204478 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:24:17.204527 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:24:17.204607 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:24:17.209694 (+ 5087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:17.210499 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:17.214684 (+ 4185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:17.214933 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:17.215040 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:17.218090 (+ 3050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:24:17.218093 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:24:17.218095 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:24:17.218120 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:24:17.220006 (+ 1886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:24:17.220668 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:24:17.220672 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:24:17.220674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:24:17.220770 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:24:17.221233 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:24:17.221236 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:24:17.221238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:24:17.221467 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:24:17.221561 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0808 22:24:17.222277 (+ 716us) server_negotiation.cc:299] Negotiation successful
0808 22:24:17.222414 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1345,"thread_start_us":102,"threads_started":1}
I20260808 22:24:17.224166 29128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:17.202787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:42379 (local address 127.9.176.193:46195)
0808 22:24:17.203250 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:17.203301 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0808 22:24:17.204155 (+ 854us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:24:17.204800 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:24:17.204806 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:24:17.205138 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:24:17.205813 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:24:17.205836 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:24:17.210699 (+ 4863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:24:17.210703 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:24:17.214541 (+ 3838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:24:17.214559 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:24:17.215107 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:24:17.215112 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:24:17.215338 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:17.216071 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:24:17.216094 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:24:17.217892 (+ 1798us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:24:17.220148 (+ 2256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:24:17.220155 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:24:17.220175 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:24:17.220546 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:24:17.220876 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:24:17.220879 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:24:17.220881 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:24:17.221117 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:24:17.221632 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:24:17.221640 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:24:17.221972 (+ 332us) client_negotiation.cc:769] Sending connection context
0808 22:24:17.223104 (+ 1132us) client_negotiation.cc:240] Negotiation successful
0808 22:24:17.223407 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":106,"threads_started":1}
I20260808 22:24:17.225425 29124 heartbeater.cc:347] Connected to a master server at 127.9.176.254:42379
I20260808 22:24:17.225683 29124 heartbeater.cc:464] Registering TS with master...
I20260808 22:24:17.226274 29124 heartbeater.cc:511] Master 127.9.176.254:42379 requested a full tablet report, sending...
I20260808 22:24:17.228103 28824 ts_manager.cc:194] Registered new tserver with Master: 9a25027701d348159e49952160a3e2f2 (127.9.176.193:36751)
I20260808 22:24:17.229645 28824 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.176.193@KRBTEST.COM'} at 127.9.176.193:46195
WARNING: no policy specified for kudu/127.9.176.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.176.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.176.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.194.keytab.
Entry for principal kudu/127.9.176.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.194.keytab.
I20260808 22:24:17.263835 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.176.212:35113/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.194:0
--local_ip_for_outbound_sockets=127.9.176.194
--webserver_interface=127.9.176.194
--webserver_port=0
--tserver_master_addrs=127.9.176.254:42379
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.194.keytab
--principal=kudu/127.9.176.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:43091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 22:24:17 dist-test-slave-ncp9 krb5kdc[28123](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786227857, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260808 22:24:17.426923 29133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:24:17.427330 29133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:24:17.427431 29133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:24:17.433385 29133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:24:17.433533 29133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:24:17.433666 29133 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.194
I20260808 22:24:17.441233 29133 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:43091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.9.176.212:35113/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.194.keytab
--principal=kudu/127.9.176.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.176.194
--webserver_port=0
--tserver_master_addrs=127.9.176.254:42379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29133
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.194
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:24:17.448063 29133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:24:17.451072 29133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:24:17.464931 29142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:24:17.465022 29140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:24:17.466554 29133 server_base.cc:1034] running on GCE node
W20260808 22:24:17.469317 29144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:24:17.469951 29133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:24:17.470768 29133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:24:17.472059 29133 hybrid_clock.cc:648] HybridClock initialized: now 1786227857471991 us; error 83 us; skew 500 ppm
Aug 08 22:24:17 dist-test-slave-ncp9 krb5kdc[28123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227857, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:24:17.475782 29133 init.cc:377] Logged in from keytab as kudu/127.9.176.194@KRBTEST.COM (short username kudu)
I20260808 22:24:17.477028 29133 webserver.cc:460] Webserver started at http://127.9.176.194:33425/ using document root <none> and password file <none>
I20260808 22:24:17.477807 29133 fs_manager.cc:359] Metadata directory not provided
I20260808 22:24:17.477953 29133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:24:17.478261 29133 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 22:24:17 dist-test-slave-ncp9 krb5kdc[28123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227857, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.194@KRBTEST.COM for HTTP/127.9.176.212@KRBTEST.COM
I20260808 22:24:17.505231 29133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ts-1/data/instance:
uuid: "5d3016f26e4843dcb074e1dd7f20d17b"
format_stamp: "Formatted at 2026-08-08 22:24:17 on dist-test-slave-ncp9"
server_key: "e24354624f4a85dca2651b1c53e25559"
server_key_iv: "27faad724efb4d0caf1bc399ca3403d2"
server_key_version: "kuduclusterkey@0"
I20260808 22:24:17.507547 29133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ts-1/wal/instance:
uuid: "5d3016f26e4843dcb074e1dd7f20d17b"
format_stamp: "Formatted at 2026-08-08 22:24:17 on dist-test-slave-ncp9"
server_key: "e24354624f4a85dca2651b1c53e25559"
server_key_iv: "27faad724efb4d0caf1bc399ca3403d2"
server_key_version: "kuduclusterkey@0"
I20260808 22:24:17.513151 29133 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260808 22:24:17.536510 29152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:24:17.538182 29133 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260808 22:24:17.538316 29133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ts-1/wal
uuid: "5d3016f26e4843dcb074e1dd7f20d17b"
format_stamp: "Formatted at 2026-08-08 22:24:17 on dist-test-slave-ncp9"
server_key: "e24354624f4a85dca2651b1c53e25559"
server_key_iv: "27faad724efb4d0caf1bc399ca3403d2"
server_key_version: "kuduclusterkey@0"
I20260808 22:24:17.538443 29133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:24:17.566816 29133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:24:17.570403 29133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:24:17.570654 29133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:24:17.571405 29133 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:24:17.572456 29133 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:24:17.572562 29133 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:24:17.572661 29133 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:24:17.572721 29133 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:24:17.586251 29133 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.194:44513
I20260808 22:24:17.587442 29133 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
I20260808 22:24:17.589452 29321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.194:44513 every 8 connection(s)
I20260808 22:24:17.594261 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 29133
I20260808 22:24:17.594385 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ts-1/wal/instance
I20260808 22:24:17.594687 9923 external_mini_cluster.cc:1428] Setting key c8697e486560aff6884f313679c87f73
Aug 08 22:24:17 dist-test-slave-ncp9 krb5kdc[28123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227857, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.194@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.176.195@KRBTEST.COM; defaulting to no policy
I20260808 22:24:17.608230 29127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:17.590359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.194:54359 (local address 127.9.176.254:42379)
0808 22:24:17.590492 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0808 22:24:17.590496 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:24:17.591583 (+ 1087us) server_negotiation.cc:407] Connection header received
0808 22:24:17.592841 (+ 1258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:24:17.592844 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:24:17.592927 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:24:17.593019 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:24:17.595146 (+ 2127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:17.595970 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:17.600089 (+ 4119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:17.600444 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:17.600551 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:17.603747 (+ 3196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:24:17.603751 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:24:17.603754 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:24:17.603783 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:24:17.605531 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:24:17.606379 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:24:17.606383 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:24:17.606385 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:24:17.606468 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:24:17.606855 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:24:17.606859 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:24:17.606860 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:24:17.607073 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:24:17.607149 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 22:24:17.607899 (+ 750us) server_negotiation.cc:299] Negotiation successful
0808 22:24:17.608048 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 22:24:17.610149 29324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:17.590869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:42379 (local address 127.9.176.194:54359)
0808 22:24:17.591412 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:17.591471 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0808 22:24:17.592535 (+ 1064us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:24:17.593204 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:24:17.593213 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:24:17.593546 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:24:17.594222 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:24:17.594242 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:24:17.596151 (+ 1909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:24:17.596154 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:24:17.599940 (+ 3786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:24:17.599953 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:24:17.600621 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:24:17.600625 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:24:17.600826 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:17.601784 (+ 958us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:24:17.601806 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:24:17.603551 (+ 1745us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:24:17.605925 (+ 2374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:24:17.605930 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:24:17.605948 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:24:17.606267 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:24:17.606585 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:24:17.606587 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:24:17.606589 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:24:17.606756 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:24:17.607232 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:24:17.607238 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:24:17.607585 (+ 347us) client_negotiation.cc:769] Sending connection context
0808 22:24:17.608423 (+ 838us) client_negotiation.cc:240] Negotiation successful
0808 22:24:17.608832 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":141,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.176.195@KRBTEST.COM" created.
I20260808 22:24:17.611609 29322 heartbeater.cc:347] Connected to a master server at 127.9.176.254:42379
I20260808 22:24:17.611881 29322 heartbeater.cc:464] Registering TS with master...
I20260808 22:24:17.612457 29322 heartbeater.cc:511] Master 127.9.176.254:42379 requested a full tablet report, sending...
I20260808 22:24:17.613703 28824 ts_manager.cc:194] Registered new tserver with Master: 5d3016f26e4843dcb074e1dd7f20d17b (127.9.176.194:44513)
I20260808 22:24:17.614461 28824 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.176.194@KRBTEST.COM'} at 127.9.176.194:54359
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.176.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.195.keytab.
Entry for principal kudu/127.9.176.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.195.keytab.
I20260808 22:24:17.643589 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.176.212:35113/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.195:0
--local_ip_for_outbound_sockets=127.9.176.195
--webserver_interface=127.9.176.195
--webserver_port=0
--tserver_master_addrs=127.9.176.254:42379
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.195.keytab
--principal=kudu/127.9.176.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:43091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 22:24:17 dist-test-slave-ncp9 krb5kdc[28123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227857, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.254@KRBTEST.COM for HTTP/127.9.176.212@KRBTEST.COM
W20260808 22:24:17.813956 29330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:24:17.814397 29330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:24:17.814500 29330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:24:17.820503 29330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:24:17.820648 29330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:24:17.820773 29330 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.195
I20260808 22:24:17.828492 29330 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:43091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.9.176.212:35113/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.195.keytab
--principal=kudu/127.9.176.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.176.195
--webserver_port=0
--tserver_master_addrs=127.9.176.254:42379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29330
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.195
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:24:17.830080 29330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:24:17.831434 29330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:24:17.844058 29340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:24:17.844606 29344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:24:17.845111 29336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:24:17.846391 29330 server_base.cc:1034] running on GCE node
I20260808 22:24:17.846838 29330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:24:17.847595 29330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:24:17.850871 29330 hybrid_clock.cc:648] HybridClock initialized: now 1786227857850795 us; error 80 us; skew 500 ppm
Aug 08 22:24:17 dist-test-slave-ncp9 krb5kdc[28123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227857, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:24:17.854949 29330 init.cc:377] Logged in from keytab as kudu/127.9.176.195@KRBTEST.COM (short username kudu)
I20260808 22:24:17.856020 29330 webserver.cc:460] Webserver started at http://127.9.176.195:46657/ using document root <none> and password file <none>
I20260808 22:24:17.856703 29330 fs_manager.cc:359] Metadata directory not provided
I20260808 22:24:17.856779 29330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:24:17.857580 29330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 22:24:17 dist-test-slave-ncp9 krb5kdc[28123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227857, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.195@KRBTEST.COM for HTTP/127.9.176.212@KRBTEST.COM
I20260808 22:24:17.871337 29330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ts-2/data/instance:
uuid: "a57a0257a8504929a3d2468f5fd6194c"
format_stamp: "Formatted at 2026-08-08 22:24:17 on dist-test-slave-ncp9"
server_key: "6fc5fc6e0f963c4d27541e6a5af7a2d6"
server_key_iv: "2aefa04addcf8f9b447fd74794e94185"
server_key_version: "kuduclusterkey@0"
I20260808 22:24:17.872140 29330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ts-2/wal/instance:
uuid: "a57a0257a8504929a3d2468f5fd6194c"
format_stamp: "Formatted at 2026-08-08 22:24:17 on dist-test-slave-ncp9"
server_key: "6fc5fc6e0f963c4d27541e6a5af7a2d6"
server_key_iv: "2aefa04addcf8f9b447fd74794e94185"
server_key_version: "kuduclusterkey@0"
I20260808 22:24:17.878566 29330 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.001s
I20260808 22:24:17.903941 29357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:24:17.905814 29330 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.003s sys 0.001s
I20260808 22:24:17.906066 29330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ts-2/wal
uuid: "a57a0257a8504929a3d2468f5fd6194c"
format_stamp: "Formatted at 2026-08-08 22:24:17 on dist-test-slave-ncp9"
server_key: "6fc5fc6e0f963c4d27541e6a5af7a2d6"
server_key_iv: "2aefa04addcf8f9b447fd74794e94185"
server_key_version: "kuduclusterkey@0"
I20260808 22:24:17.906265 29330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:24:17.957146 29330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:24:17.960812 29330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:24:17.961035 29330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:24:17.961768 29330 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:24:17.962826 29330 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:24:17.962889 29330 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:24:17.962944 29330 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:24:17.962971 29330 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:24:17.974325 29330 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.195:43543
I20260808 22:24:17.974447 29518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.195:43543 every 8 connection(s)
I20260808 22:24:17.975512 29330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
I20260808 22:24:17.977183 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 29330
I20260808 22:24:17.977316 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ts-2/wal/instance
I20260808 22:24:17.977620 9923 external_mini_cluster.cc:1428] Setting key 45efd64425bc16670d7e344070dd88fc
Aug 08 22:24:17 dist-test-slave-ncp9 krb5kdc[28123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227857, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.195@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
I20260808 22:24:17.993307 29127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:17.980111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.195:34537 (local address 127.9.176.254:42379)
0808 22:24:17.980227 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0808 22:24:17.980231 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:24:17.980246 (+ 15us) server_negotiation.cc:407] Connection header received
0808 22:24:17.981113 (+ 867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:24:17.981117 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:24:17.981170 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:24:17.981223 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:24:17.982904 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:17.983754 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:17.985472 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:17.985823 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:17.985927 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:17.988618 (+ 2691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:24:17.988622 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:24:17.988625 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:24:17.988653 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:24:17.990225 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:24:17.990794 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:24:17.990798 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:24:17.990800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:24:17.990895 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:24:17.991270 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:24:17.991274 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:24:17.991276 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:24:17.991531 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:24:17.991643 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0808 22:24:17.993041 (+ 1398us) server_negotiation.cc:299] Negotiation successful
0808 22:24:17.993159 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 22:24:17.994803 29521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:17.979083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:42379 (local address 127.9.176.195:34537)
0808 22:24:17.979697 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:17.979758 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0808 22:24:17.980831 (+ 1073us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:24:17.981667 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:24:17.981676 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:24:17.982021 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:24:17.982731 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:24:17.982754 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:24:17.983900 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:24:17.983904 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:24:17.985339 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:24:17.985352 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:24:17.985980 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:24:17.985985 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:24:17.986178 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:17.986794 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:24:17.986814 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:24:17.988437 (+ 1623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:24:17.990352 (+ 1915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:24:17.990358 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:24:17.990372 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:24:17.990685 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:24:17.990994 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:24:17.990997 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:24:17.991006 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:24:17.991170 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:24:17.991889 (+ 719us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:24:17.991896 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:24:17.992331 (+ 435us) client_negotiation.cc:769] Sending connection context
0808 22:24:17.993539 (+ 1208us) client_negotiation.cc:240] Negotiation successful
0808 22:24:17.993847 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":164,"threads_started":1}
I20260808 22:24:17.995990 29519 heartbeater.cc:347] Connected to a master server at 127.9.176.254:42379
I20260808 22:24:17.996249 29519 heartbeater.cc:464] Registering TS with master...
I20260808 22:24:17.996840 29519 heartbeater.cc:511] Master 127.9.176.254:42379 requested a full tablet report, sending...
I20260808 22:24:17.998015 28824 ts_manager.cc:194] Registered new tserver with Master: a57a0257a8504929a3d2468f5fd6194c (127.9.176.195:43543)
I20260808 22:24:17.998697 28824 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.176.195@KRBTEST.COM'} at 127.9.176.195:34537
I20260808 22:24:18.005038 9923 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 22:24:18.005966 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 28936
I20260808 22:24:18.040132 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 29133
I20260808 22:24:18.053475 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 29330
I20260808 22:24:18.065092 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 28783
I20260808 22:24:18.086113 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:42379
--webserver_interface=127.9.176.254
--webserver_port=44953
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab
--principal=kudu/127.9.176.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:43091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.176.254:42379
--ranger_config_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.176.212:35113/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.9.176.212:35113/kms with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:24:18.194171 29524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:24:18.194430 29524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:24:18.194468 29524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:24:18.198410 29524 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:24:18.198465 29524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:24:18.198483 29524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:24:18.198498 29524 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:24:18.198515 29524 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:24:18.203830 29524 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:43091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.9.176.212:35113/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.9.176.254:42379
--ranger_config_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab
--principal=kudu/127.9.176.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:42379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.176.254
--webserver_port=44953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29524
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:24:18.204775 29524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:24:18.205744 29524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:24:18.213738 29534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:24:18.213757 29530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:24:18.213760 29531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:24:18.213959 29524 server_base.cc:1034] running on GCE node
I20260808 22:24:18.214586 29524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:24:18.215567 29524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:24:18.216727 29524 hybrid_clock.cc:648] HybridClock initialized: now 1786227858216709 us; error 28 us; skew 500 ppm
Aug 08 22:24:18 dist-test-slave-ncp9 krb5kdc[28123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227858, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:24:18.219408 29524 init.cc:377] Logged in from keytab as kudu/127.9.176.254@KRBTEST.COM (short username kudu)
I20260808 22:24:18.220256 29524 webserver.cc:460] Webserver started at http://127.9.176.254:44953/ using document root <none> and password file <none>
I20260808 22:24:18.220762 29524 fs_manager.cc:359] Metadata directory not provided
I20260808 22:24:18.220834 29524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 08 22:24:18 dist-test-slave-ncp9 krb5kdc[28123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227858, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.254@KRBTEST.COM for HTTP/127.9.176.212@KRBTEST.COM
I20260808 22:24:18.258050 29524 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260808 22:24:18.260278 29545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:24:18.261556 29524 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 22:24:18.261679 29524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/master-0/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/master-0/wal
uuid: "ece099de893c46439733749788a8d611"
format_stamp: "Formatted at 2026-08-08 22:24:16 on dist-test-slave-ncp9"
server_key: "25516f32aae133bf91ed449ea49cbade"
server_key_iv: "75590ebbbdf2a6f8cfc27b4307ee0688"
server_key_version: "kuduclusterkey@0"
I20260808 22:24:18.262235 29524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:24:18.290565 29524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:24:18.293038 29524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:24:18.293213 29524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:24:18.300113 29524 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.254:42379
I20260808 22:24:18.300134 29627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.254:42379 every 8 connection(s)
I20260808 22:24:18.301167 29524 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
I20260808 22:24:18.303284 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 29524
I20260808 22:24:18.303668 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.176.212:35113/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.193:36751
--local_ip_for_outbound_sockets=127.9.176.193
--tserver_master_addrs=127.9.176.254:42379
--webserver_port=42917
--webserver_interface=127.9.176.193
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.193.keytab
--principal=kudu/127.9.176.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:43091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.9.176.212:35113/kms with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 22:24:18.305402 29628 sys_catalog.cc:263] Verifying existing consensus state
I20260808 22:24:18.307183 29628 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ece099de893c46439733749788a8d611: Bootstrap starting.
I20260808 22:24:18.327991 29628 log.cc:824] T 00000000000000000000000000000000 P ece099de893c46439733749788a8d611: Log is configured to *not* fsync() on all Append() calls
I20260808 22:24:18.334477 29628 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ece099de893c46439733749788a8d611: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 22:24:18.334838 29628 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ece099de893c46439733749788a8d611: Bootstrap complete.
I20260808 22:24:18.338686 29628 raft_consensus.cc:348] T 00000000000000000000000000000000 P ece099de893c46439733749788a8d611 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ece099de893c46439733749788a8d611" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 42379 } }
I20260808 22:24:18.339057 29628 raft_consensus.cc:729] T 00000000000000000000000000000000 P ece099de893c46439733749788a8d611 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ece099de893c46439733749788a8d611, State: Initialized, Role: FOLLOWER
I20260808 22:24:18.339527 29628 consensus_queue.cc:260] T 00000000000000000000000000000000 P ece099de893c46439733749788a8d611 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ece099de893c46439733749788a8d611" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 42379 } }
I20260808 22:24:18.339646 29628 raft_consensus.cc:388] T 00000000000000000000000000000000 P ece099de893c46439733749788a8d611 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:24:18.339705 29628 raft_consensus.cc:482] T 00000000000000000000000000000000 P ece099de893c46439733749788a8d611 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:24:18.339797 29628 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ece099de893c46439733749788a8d611 [term 1 FOLLOWER]: Advancing to term 2
I20260808 22:24:18.341598 29628 raft_consensus.cc:504] T 00000000000000000000000000000000 P ece099de893c46439733749788a8d611 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ece099de893c46439733749788a8d611" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 42379 } }
I20260808 22:24:18.341897 29628 leader_election.cc:302] T 00000000000000000000000000000000 P ece099de893c46439733749788a8d611 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ece099de893c46439733749788a8d611; no voters:
I20260808 22:24:18.342195 29628 leader_election.cc:288] T 00000000000000000000000000000000 P ece099de893c46439733749788a8d611 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260808 22:24:18.342283 29634 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ece099de893c46439733749788a8d611 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 22:24:18.342474 29634 raft_consensus.cc:686] T 00000000000000000000000000000000 P ece099de893c46439733749788a8d611 [term 2 LEADER]: Becoming Leader. State: Replica: ece099de893c46439733749788a8d611, State: Running, Role: LEADER
I20260808 22:24:18.342767 29634 consensus_queue.cc:237] T 00000000000000000000000000000000 P ece099de893c46439733749788a8d611 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ece099de893c46439733749788a8d611" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 42379 } }
I20260808 22:24:18.343379 29628 sys_catalog.cc:564] T 00000000000000000000000000000000 P ece099de893c46439733749788a8d611 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:24:18.343698 29634 sys_catalog.cc:455] T 00000000000000000000000000000000 P ece099de893c46439733749788a8d611 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ece099de893c46439733749788a8d611. Latest consensus state: current_term: 2 leader_uuid: "ece099de893c46439733749788a8d611" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ece099de893c46439733749788a8d611" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 42379 } } }
I20260808 22:24:18.343889 29634 sys_catalog.cc:458] T 00000000000000000000000000000000 P ece099de893c46439733749788a8d611 [sys.catalog]: This master's current role is: LEADER
I20260808 22:24:18.344542 29636 sys_catalog.cc:455] T 00000000000000000000000000000000 P ece099de893c46439733749788a8d611 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ece099de893c46439733749788a8d611" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ece099de893c46439733749788a8d611" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 42379 } } }
I20260808 22:24:18.344697 29636 sys_catalog.cc:458] T 00000000000000000000000000000000 P ece099de893c46439733749788a8d611 [sys.catalog]: This master's current role is: LEADER
I20260808 22:24:18.345194 29640 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:24:18.345607 29628 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260808 22:24:18.351488 29640 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:24:18.352231 29640 catalog_manager.cc:1259] Loaded cluster ID: 99eb7227a6e64a5a8a574dec6b0afe3d
I20260808 22:24:18.352283 29640 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:24:18.354915 29640 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:24:18.355631 29640 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ece099de893c46439733749788a8d611: Loaded TSK: 0
I20260808 22:24:18.356884 29640 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 22:24:18.452121 29632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:24:18.452435 29632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:24:18.452507 29632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:24:18.458366 29632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:24:18.458443 29632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:24:18.458529 29632 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.193
I20260808 22:24:18.466053 29632 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:43091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.9.176.212:35113/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.193.keytab
--principal=kudu/127.9.176.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.193:36751
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.176.193
--webserver_port=42917
--tserver_master_addrs=127.9.176.254:42379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.193
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:24:18.467338 29632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:24:18.468641 29632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:24:18.481593 29683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:24:18.481819 29632 server_base.cc:1034] running on GCE node
W20260808 22:24:18.482107 29687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:24:18.482478 29682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:24:18.483028 29632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:24:18.483783 29632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:24:18.484959 29632 hybrid_clock.cc:648] HybridClock initialized: now 1786227858484930 us; error 44 us; skew 500 ppm
Aug 08 22:24:18 dist-test-slave-ncp9 krb5kdc[28123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227858, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:24:18.488556 29632 init.cc:377] Logged in from keytab as kudu/127.9.176.193@KRBTEST.COM (short username kudu)
I20260808 22:24:18.489753 29632 webserver.cc:460] Webserver started at http://127.9.176.193:42917/ using document root <none> and password file <none>
I20260808 22:24:18.490486 29632 fs_manager.cc:359] Metadata directory not provided
I20260808 22:24:18.490651 29632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 08 22:24:18 dist-test-slave-ncp9 krb5kdc[28123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227858, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.193@KRBTEST.COM for HTTP/127.9.176.212@KRBTEST.COM
I20260808 22:24:18.530486 29632 fs_manager.cc:709] Time spent opening directory manager: real 0.019s user 0.003s sys 0.000s
I20260808 22:24:18.537158 29704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:24:18.542173 29632 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.001s sys 0.000s
I20260808 22:24:18.542348 29632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ts-0/wal
uuid: "9a25027701d348159e49952160a3e2f2"
format_stamp: "Formatted at 2026-08-08 22:24:17 on dist-test-slave-ncp9"
server_key: "04dc315f4cb42833148e6ea486a2d39e"
server_key_iv: "b1e68456936f37514602129ec02440ff"
server_key_version: "kuduclusterkey@0"
I20260808 22:24:18.543135 29632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:24:18.561604 29632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:24:18.565508 29632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:24:18.565719 29632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:24:18.566541 29632 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:24:18.567646 29632 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:24:18.567711 29632 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:24:18.567767 29632 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:24:18.567797 29632 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:24:18.579327 29632 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.193:36751
I20260808 22:24:18.580703 29632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
I20260808 22:24:18.581656 29864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.193:36751 every 8 connection(s)
I20260808 22:24:18.583007 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 29632
I20260808 22:24:18.583400 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.176.212:35113/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.194:44513
--local_ip_for_outbound_sockets=127.9.176.194
--tserver_master_addrs=127.9.176.254:42379
--webserver_port=33425
--webserver_interface=127.9.176.194
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.194.keytab
--principal=kudu/127.9.176.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:43091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.9.176.212:35113/kms with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 22:24:18 dist-test-slave-ncp9 krb5kdc[28123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227858, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.193@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
I20260808 22:24:18.606645 29870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:18.585041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:42379 (local address 127.9.176.193:47035)
0808 22:24:18.585630 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:18.585691 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0808 22:24:18.586632 (+ 941us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:24:18.588232 (+ 1600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:24:18.588240 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:24:18.589050 (+ 810us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:24:18.589667 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:24:18.589688 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:24:18.595163 (+ 5475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:24:18.595167 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:24:18.596656 (+ 1489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:24:18.596671 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:24:18.598070 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:24:18.598074 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:24:18.598271 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:18.598995 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:24:18.599029 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:24:18.600536 (+ 1507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:24:18.603159 (+ 2623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:24:18.603165 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:24:18.603180 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:24:18.603494 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:24:18.603865 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:24:18.603868 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:24:18.603870 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:24:18.604041 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:24:18.604769 (+ 728us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:24:18.604775 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:24:18.605094 (+ 319us) client_negotiation.cc:769] Sending connection context
0808 22:24:18.605283 (+ 189us) client_negotiation.cc:240] Negotiation successful
0808 22:24:18.605596 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":158,"threads_started":1}
I20260808 22:24:18.608146 29871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:18.586592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.193:47035 (local address 127.9.176.254:42379)
0808 22:24:18.587112 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0808 22:24:18.587130 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0808 22:24:18.587167 (+ 37us) server_negotiation.cc:407] Connection header received
0808 22:24:18.587390 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:24:18.587401 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:24:18.587816 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:24:18.588142 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:24:18.589863 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:18.594976 (+ 5113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:18.596806 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:18.597627 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:18.597807 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:18.600752 (+ 2945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:24:18.600759 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:24:18.600776 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:24:18.600808 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:24:18.603013 (+ 2205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:24:18.603623 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:24:18.603629 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:24:18.603635 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:24:18.603767 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:24:18.604141 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:24:18.604144 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:24:18.604146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:24:18.604592 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:24:18.604705 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0808 22:24:18.606987 (+ 2282us) server_negotiation.cc:299] Negotiation successful
0808 22:24:18.607283 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":123,"threads_started":1}
I20260808 22:24:18.611403 29865 heartbeater.cc:347] Connected to a master server at 127.9.176.254:42379
I20260808 22:24:18.611690 29865 heartbeater.cc:464] Registering TS with master...
I20260808 22:24:18.612303 29865 heartbeater.cc:511] Master 127.9.176.254:42379 requested a full tablet report, sending...
W20260808 22:24:18.614710 29865 heartbeater.cc:644] Failed to heartbeat to 127.9.176.254:42379 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260808 22:24:18.614871 29865 heartbeater.cc:464] Registering TS with master...
I20260808 22:24:18.615072 29865 heartbeater.cc:511] Master 127.9.176.254:42379 requested a full tablet report, sending...
I20260808 22:24:18.615988 29865 heartbeater.cc:464] Registering TS with master...
I20260808 22:24:18.616190 29865 heartbeater.cc:511] Master 127.9.176.254:42379 requested a full tablet report, sending...
W20260808 22:24:18.617015 29865 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 22:24:18.723954 29867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:24:18.724336 29867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:24:18.724404 29867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:24:18.729357 29867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:24:18.729429 29867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:24:18.729516 29867 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.194
I20260808 22:24:18.736529 29867 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:43091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.9.176.212:35113/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.194.keytab
--principal=kudu/127.9.176.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.194:44513
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.176.194
--webserver_port=33425
--tserver_master_addrs=127.9.176.254:42379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29867
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.194
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:24:18.737947 29867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:24:18.739342 29867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:24:18.749722 29879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:24:18.750078 29885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:24:18.750134 29867 server_base.cc:1034] running on GCE node
W20260808 22:24:18.749814 29881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:24:18.752827 29867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:24:18.753669 29867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:24:18.754851 29867 hybrid_clock.cc:648] HybridClock initialized: now 1786227858754795 us; error 73 us; skew 500 ppm
Aug 08 22:24:18 dist-test-slave-ncp9 krb5kdc[28123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227858, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:24:18.758437 29867 init.cc:377] Logged in from keytab as kudu/127.9.176.194@KRBTEST.COM (short username kudu)
I20260808 22:24:18.759470 29867 webserver.cc:460] Webserver started at http://127.9.176.194:33425/ using document root <none> and password file <none>
I20260808 22:24:18.760429 29867 fs_manager.cc:359] Metadata directory not provided
I20260808 22:24:18.760608 29867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 08 22:24:18 dist-test-slave-ncp9 krb5kdc[28123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227858, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.194@KRBTEST.COM for HTTP/127.9.176.212@KRBTEST.COM
I20260808 22:24:18.791162 29867 fs_manager.cc:709] Time spent opening directory manager: real 0.014s user 0.003s sys 0.000s
I20260808 22:24:18.793774 29896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:24:18.795523 29867 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 22:24:18.795719 29867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ts-1/wal
uuid: "5d3016f26e4843dcb074e1dd7f20d17b"
format_stamp: "Formatted at 2026-08-08 22:24:17 on dist-test-slave-ncp9"
server_key: "e24354624f4a85dca2651b1c53e25559"
server_key_iv: "27faad724efb4d0caf1bc399ca3403d2"
server_key_version: "kuduclusterkey@0"
I20260808 22:24:18.796561 29867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:24:18.820363 29867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:24:18.823979 29867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:24:18.824167 29867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:24:18.824761 29867 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:24:18.825661 29867 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:24:18.825711 29867 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:24:18.825764 29867 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:24:18.825788 29867 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:24:18.835867 29867 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.194:44513
I20260808 22:24:18.836033 30057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.194:44513 every 8 connection(s)
I20260808 22:24:18.837193 29867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
I20260808 22:24:18.847083 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 29867
I20260808 22:24:18.847599 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.176.212:35113/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.195:43543
--local_ip_for_outbound_sockets=127.9.176.195
--tserver_master_addrs=127.9.176.254:42379
--webserver_port=46657
--webserver_interface=127.9.176.195
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.195.keytab
--principal=kudu/127.9.176.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:43091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.9.176.212:35113/kms with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 22:24:18 dist-test-slave-ncp9 krb5kdc[28123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227858, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.194@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
I20260808 22:24:18.853562 29871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:18.840752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.194:38351 (local address 127.9.176.254:42379)
0808 22:24:18.840921 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0808 22:24:18.840926 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:24:18.841887 (+ 961us) server_negotiation.cc:407] Connection header received
0808 22:24:18.842811 (+ 924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:24:18.842815 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:24:18.842872 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:24:18.842953 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:24:18.844516 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:18.845454 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:18.846901 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:18.847137 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:18.847231 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:18.849828 (+ 2597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:24:18.849832 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:24:18.849835 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:24:18.849865 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:24:18.851468 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:24:18.852091 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:24:18.852094 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:24:18.852096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:24:18.852181 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:24:18.852489 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:24:18.852493 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:24:18.852495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:24:18.852680 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:24:18.852765 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0808 22:24:18.853330 (+ 565us) server_negotiation.cc:299] Negotiation successful
0808 22:24:18.853439 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 22:24:18.854878 30061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:18.839652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:42379 (local address 127.9.176.194:38351)
0808 22:24:18.841728 (+ 2076us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:18.841769 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 22:24:18.842599 (+ 830us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:24:18.843260 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:24:18.843270 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:24:18.843628 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:24:18.844288 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:24:18.844311 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:24:18.845621 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:24:18.845624 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:24:18.846777 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:24:18.846787 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:24:18.847381 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:24:18.847384 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:24:18.847523 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:18.848110 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:24:18.848126 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:24:18.849629 (+ 1503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:24:18.851598 (+ 1969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:24:18.851604 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:24:18.851619 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:24:18.851951 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:24:18.852263 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:24:18.852266 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:24:18.852268 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:24:18.852401 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:24:18.852779 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:24:18.852785 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:24:18.853075 (+ 290us) client_negotiation.cc:769] Sending connection context
0808 22:24:18.853781 (+ 706us) client_negotiation.cc:240] Negotiation successful
0808 22:24:18.854109 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1890,"thread_start_us":135,"threads_started":1}
I20260808 22:24:18.855973 30058 heartbeater.cc:347] Connected to a master server at 127.9.176.254:42379
I20260808 22:24:18.856213 30058 heartbeater.cc:464] Registering TS with master...
I20260808 22:24:18.856680 30058 heartbeater.cc:511] Master 127.9.176.254:42379 requested a full tablet report, sending...
W20260808 22:24:18.858165 30058 heartbeater.cc:644] Failed to heartbeat to 127.9.176.254:42379 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260808 22:24:18.858281 30058 heartbeater.cc:464] Registering TS with master...
I20260808 22:24:18.858428 30058 heartbeater.cc:511] Master 127.9.176.254:42379 requested a full tablet report, sending...
I20260808 22:24:18.859231 30058 heartbeater.cc:464] Registering TS with master...
I20260808 22:24:18.859383 30058 heartbeater.cc:511] Master 127.9.176.254:42379 requested a full tablet report, sending...
W20260808 22:24:18.860242 30058 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 22:24:18.996418 30062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:24:18.996778 30062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:24:18.996841 30062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:24:19.002374 30062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:24:19.002446 30062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:24:19.002525 30062 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.195
I20260808 22:24:19.008059 30062 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:43091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.9.176.212:35113/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.195.keytab
--principal=kudu/127.9.176.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.195:43543
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.176.195
--webserver_port=46657
--tserver_master_addrs=127.9.176.254:42379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30062
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.195
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:24:19.009497 30062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:24:19.010815 30062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:24:19.022531 30070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:24:19.022866 30062 server_base.cc:1034] running on GCE node
W20260808 22:24:19.022614 30072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:24:19.023190 30074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:24:19.023658 30062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:24:19.024454 30062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:24:19.025641 30062 hybrid_clock.cc:648] HybridClock initialized: now 1786227859025608 us; error 48 us; skew 500 ppm
Aug 08 22:24:19 dist-test-slave-ncp9 krb5kdc[28123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227859, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:24:19.028826 30062 init.cc:377] Logged in from keytab as kudu/127.9.176.195@KRBTEST.COM (short username kudu)
I20260808 22:24:19.029762 30062 webserver.cc:460] Webserver started at http://127.9.176.195:46657/ using document root <none> and password file <none>
I20260808 22:24:19.030313 30062 fs_manager.cc:359] Metadata directory not provided
I20260808 22:24:19.030416 30062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 08 22:24:19 dist-test-slave-ncp9 krb5kdc[28123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227859, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.195@KRBTEST.COM for HTTP/127.9.176.212@KRBTEST.COM
I20260808 22:24:19.048435 30062 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260808 22:24:19.051190 30083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:24:19.052467 30062 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260808 22:24:19.052594 30062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ts-2/wal
uuid: "a57a0257a8504929a3d2468f5fd6194c"
format_stamp: "Formatted at 2026-08-08 22:24:17 on dist-test-slave-ncp9"
server_key: "6fc5fc6e0f963c4d27541e6a5af7a2d6"
server_key_iv: "2aefa04addcf8f9b447fd74794e94185"
server_key_version: "kuduclusterkey@0"
I20260808 22:24:19.053182 30062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:24:19.070317 30062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:24:19.073432 30062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:24:19.073618 30062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:24:19.074195 30062 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:24:19.075143 30062 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:24:19.075199 30062 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:24:19.075248 30062 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:24:19.075268 30062 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:24:19.089380 30062 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.195:43543
I20260808 22:24:19.089493 30260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.195:43543 every 8 connection(s)
I20260808 22:24:19.090502 30062 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
Aug 08 22:24:19 dist-test-slave-ncp9 krb5kdc[28123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227859, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.195@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
I20260808 22:24:19.100333 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 30062
I20260808 22:24:19.105433 30263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:19.092848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:42379 (local address 127.9.176.195:56653)
0808 22:24:19.093363 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:19.093416 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0808 22:24:19.094270 (+ 854us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:24:19.094813 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:24:19.094820 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:24:19.095093 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:24:19.095581 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:24:19.095595 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:24:19.096371 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:24:19.096375 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:24:19.097306 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:24:19.097317 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:24:19.097796 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:24:19.097801 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:24:19.097969 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:19.098512 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:24:19.098528 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:24:19.100467 (+ 1939us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:24:19.102538 (+ 2071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:24:19.102544 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:24:19.102556 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:24:19.102888 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:24:19.103178 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:24:19.103181 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:24:19.103183 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:24:19.103322 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:24:19.103749 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:24:19.103756 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:24:19.104060 (+ 304us) client_negotiation.cc:769] Sending connection context
0808 22:24:19.104258 (+ 198us) client_negotiation.cc:240] Negotiation successful
0808 22:24:19.104598 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":130,"threads_started":1}
I20260808 22:24:19.106118 29871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:19.092432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.195:56653 (local address 127.9.176.254:42379)
0808 22:24:19.092581 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0808 22:24:19.092586 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:24:19.093543 (+ 957us) server_negotiation.cc:407] Connection header received
0808 22:24:19.094487 (+ 944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:24:19.094491 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:24:19.094549 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:24:19.094624 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:24:19.095725 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:19.096251 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:19.097423 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:19.097658 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:19.097748 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:19.100681 (+ 2933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:24:19.100685 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:24:19.100688 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:24:19.100719 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:24:19.102422 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:24:19.103002 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:24:19.103005 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:24:19.103007 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:24:19.103088 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:24:19.103415 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:24:19.103418 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:24:19.103420 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:24:19.103646 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:24:19.103738 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0808 22:24:19.105808 (+ 2070us) server_negotiation.cc:299] Negotiation successful
0808 22:24:19.105917 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 22:24:19.107203 30261 heartbeater.cc:347] Connected to a master server at 127.9.176.254:42379
I20260808 22:24:19.107448 30261 heartbeater.cc:464] Registering TS with master...
I20260808 22:24:19.107964 30261 heartbeater.cc:511] Master 127.9.176.254:42379 requested a full tablet report, sending...
I20260808 22:24:19.109599 29578 ts_manager.cc:194] Registered new tserver with Master: a57a0257a8504929a3d2468f5fd6194c (127.9.176.195:43543)
I20260808 22:24:19.110960 29578 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.176.195@KRBTEST.COM'} at 127.9.176.195:56653
I20260808 22:24:19.112740 30265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:19.101359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54412 (local address 127.9.176.254:42379)
0808 22:24:19.101616 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0808 22:24:19.101624 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 22:24:19.102638 (+ 1014us) server_negotiation.cc:407] Connection header received
0808 22:24:19.102782 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:24:19.102785 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:24:19.102837 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:24:19.102925 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:24:19.103835 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:19.104575 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:19.105940 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:19.106281 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:19.106378 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:19.108271 (+ 1893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:24:19.108275 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:24:19.108277 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:24:19.108301 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:24:19.110182 (+ 1881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:24:19.110664 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:24:19.110668 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:24:19.110669 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:24:19.110752 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:24:19.111012 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:24:19.111015 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:24:19.111017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:24:19.111340 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:24:19.111516 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0808 22:24:19.112460 (+ 944us) server_negotiation.cc:299] Negotiation successful
0808 22:24:19.112583 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":97,"threads_started":1}
Aug 08 22:24:19 dist-test-slave-ncp9 krb5kdc[28123](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786227859, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:24:19.617256 29865 heartbeater.cc:464] Registering TS with master...
I20260808 22:24:19.617580 29865 heartbeater.cc:511] Master 127.9.176.254:42379 requested a full tablet report, sending...
I20260808 22:24:19.618703 29578 ts_manager.cc:194] Registered new tserver with Master: 9a25027701d348159e49952160a3e2f2 (127.9.176.193:36751)
I20260808 22:24:19.620532 29578 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.176.193@KRBTEST.COM'} at 127.9.176.193:47035
Aug 08 22:24:19 dist-test-slave-ncp9 krb5kdc[28123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227859, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.254@KRBTEST.COM for HTTP/127.9.176.212@KRBTEST.COM
I20260808 22:24:19.860487 30058 heartbeater.cc:464] Registering TS with master...
I20260808 22:24:19.860805 30058 heartbeater.cc:511] Master 127.9.176.254:42379 requested a full tablet report, sending...
I20260808 22:24:19.861511 29578 ts_manager.cc:194] Registered new tserver with Master: 5d3016f26e4843dcb074e1dd7f20d17b (127.9.176.194:44513)
I20260808 22:24:19.862103 29578 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.176.194@KRBTEST.COM'} at 127.9.176.194:38351
I20260808 22:24:19.867426 9923 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 22:24:19.882915 30278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:19.869069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54422 (local address 127.9.176.254:42379)
0808 22:24:19.869652 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0808 22:24:19.869657 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:24:19.869670 (+ 13us) server_negotiation.cc:407] Connection header received
0808 22:24:19.869705 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:24:19.869708 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:24:19.869753 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:24:19.869837 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:24:19.870724 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:19.871536 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:19.872763 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:19.873140 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:19.873244 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:19.879191 (+ 5947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:24:19.879195 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:24:19.879198 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:24:19.879230 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:24:19.881035 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:24:19.881571 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:24:19.881574 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:24:19.881576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:24:19.881671 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:24:19.881957 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:24:19.881960 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:24:19.881962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:24:19.882213 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:24:19.882340 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0808 22:24:19.882605 (+ 265us) server_negotiation.cc:299] Negotiation successful
0808 22:24:19.882742 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":88,"threads_started":1}
I20260808 22:24:19.887557 29578 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54422:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 22:24:19.892324 29578 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 22:24:19.912602 30286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:19.904249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.193:36751 (local address 127.0.0.1:53134)
0808 22:24:19.906586 (+ 2337us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:19.906639 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0808 22:24:19.906767 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:24:19.907535 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:24:19.907543 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:24:19.907569 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:24:19.907722 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:24:19.907730 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:24:19.909324 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:24:19.909327 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:24:19.910672 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:24:19.910683 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:24:19.912074 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:24:19.912078 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:24:19.912182 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:19.912221 (+ 39us) client_negotiation.cc:769] Sending connection context
0808 22:24:19.912321 (+ 100us) client_negotiation.cc:240] Negotiation successful
0808 22:24:19.912432 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2166,"thread_start_us":116,"threads_started":1}
I20260808 22:24:19.913245 30284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:19.904382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53134 (local address 127.9.176.193:36751)
0808 22:24:19.907061 (+ 2679us) server_negotiation.cc:206] Beginning negotiation
0808 22:24:19.907068 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 22:24:19.907091 (+ 23us) server_negotiation.cc:407] Connection header received
0808 22:24:19.907168 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:24:19.907173 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:24:19.907376 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:24:19.907529 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:24:19.907868 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:19.909186 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:19.910807 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:19.911750 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:19.911828 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:19.911969 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0808 22:24:19.912986 (+ 1017us) server_negotiation.cc:299] Negotiation successful
0808 22:24:19.913103 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2578,"thread_start_us":100,"threads_started":1}
I20260808 22:24:19.917757 29769 tablet_service.cc:1467] Processing CreateTablet for tablet 4f130ab719184374966292ad207913fc (DEFAULT_TABLE table=test-table [id=455aedb11db349298f44f9c5328b89be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:24:19.919021 29769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f130ab719184374966292ad207913fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:24:19.919461 30290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:19.906802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44226 (local address 127.9.176.195:43543)
0808 22:24:19.907095 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0808 22:24:19.907102 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 22:24:19.914102 (+ 7000us) server_negotiation.cc:407] Connection header received
0808 22:24:19.914189 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:24:19.914195 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:24:19.914389 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:24:19.914531 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:24:19.914904 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:19.916090 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:19.917570 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:19.918515 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:19.918617 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:19.918763 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0808 22:24:19.919214 (+ 451us) server_negotiation.cc:299] Negotiation successful
0808 22:24:19.919316 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":102,"threads_started":1}
I20260808 22:24:19.919929 30286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:19.906390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.195:43543 (local address 127.0.0.1:44226)
0808 22:24:19.913910 (+ 7520us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:19.913924 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 22:24:19.913993 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:24:19.914680 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:24:19.914683 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:24:19.914700 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:24:19.914770 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:24:19.914776 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:24:19.916252 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:24:19.916256 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:24:19.917426 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:24:19.917438 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:24:19.918826 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:24:19.918829 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:24:19.918922 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:19.918949 (+ 27us) client_negotiation.cc:769] Sending connection context
0808 22:24:19.919018 (+ 69us) client_negotiation.cc:240] Negotiation successful
0808 22:24:19.919803 (+ 785us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7469,"thread_start_us":102,"threads_started":1}
I20260808 22:24:19.923148 30145 tablet_service.cc:1467] Processing CreateTablet for tablet 4f130ab719184374966292ad207913fc (DEFAULT_TABLE table=test-table [id=455aedb11db349298f44f9c5328b89be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:24:19.924273 30145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f130ab719184374966292ad207913fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:24:19.928784 30287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:19.905444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55462 (local address 127.9.176.194:44513)
0808 22:24:19.905742 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0808 22:24:19.905750 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 22:24:19.907879 (+ 2129us) server_negotiation.cc:407] Connection header received
0808 22:24:19.908718 (+ 839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:24:19.908724 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:24:19.908988 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:24:19.909114 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:24:19.909562 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:19.910704 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:19.912986 (+ 2282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:19.922514 (+ 9528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:19.922602 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:19.922745 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0808 22:24:19.928497 (+ 5752us) server_negotiation.cc:299] Negotiation successful
0808 22:24:19.928629 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":92,"threads_started":1}
I20260808 22:24:19.929252 30289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:19.904885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.194:44513 (local address 127.0.0.1:55462)
0808 22:24:19.907779 (+ 2894us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:19.907792 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 22:24:19.907960 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:24:19.909317 (+ 1357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:24:19.909320 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:24:19.909338 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:24:19.909431 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:24:19.909437 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:24:19.910853 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:24:19.910856 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:24:19.912805 (+ 1949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:24:19.912817 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:24:19.928152 (+ 15335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:24:19.928156 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:24:19.928287 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:19.928318 (+ 31us) client_negotiation.cc:769] Sending connection context
0808 22:24:19.929018 (+ 700us) client_negotiation.cc:240] Negotiation successful
0808 22:24:19.929114 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2805,"thread_start_us":96,"threads_started":1}
I20260808 22:24:19.932261 29961 tablet_service.cc:1467] Processing CreateTablet for tablet 4f130ab719184374966292ad207913fc (DEFAULT_TABLE table=test-table [id=455aedb11db349298f44f9c5328b89be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:24:19.933369 29961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f130ab719184374966292ad207913fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:24:19.933665 30292 tablet_bootstrap.cc:492] T 4f130ab719184374966292ad207913fc P a57a0257a8504929a3d2468f5fd6194c: Bootstrap starting.
I20260808 22:24:19.936062 30292 tablet_bootstrap.cc:654] T 4f130ab719184374966292ad207913fc P a57a0257a8504929a3d2468f5fd6194c: Neither blocks nor log segments found. Creating new log.
I20260808 22:24:19.937028 30292 log.cc:824] T 4f130ab719184374966292ad207913fc P a57a0257a8504929a3d2468f5fd6194c: Log is configured to *not* fsync() on all Append() calls
I20260808 22:24:19.937847 30291 tablet_bootstrap.cc:492] T 4f130ab719184374966292ad207913fc P 9a25027701d348159e49952160a3e2f2: Bootstrap starting.
I20260808 22:24:19.940140 30291 tablet_bootstrap.cc:654] T 4f130ab719184374966292ad207913fc P 9a25027701d348159e49952160a3e2f2: Neither blocks nor log segments found. Creating new log.
I20260808 22:24:19.941020 30291 log.cc:824] T 4f130ab719184374966292ad207913fc P 9a25027701d348159e49952160a3e2f2: Log is configured to *not* fsync() on all Append() calls
I20260808 22:24:19.943782 30292 tablet_bootstrap.cc:492] T 4f130ab719184374966292ad207913fc P a57a0257a8504929a3d2468f5fd6194c: No bootstrap required, opened a new log
I20260808 22:24:19.944015 30292 ts_tablet_manager.cc:1397] T 4f130ab719184374966292ad207913fc P a57a0257a8504929a3d2468f5fd6194c: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260808 22:24:19.944236 30291 tablet_bootstrap.cc:492] T 4f130ab719184374966292ad207913fc P 9a25027701d348159e49952160a3e2f2: No bootstrap required, opened a new log
I20260808 22:24:19.944445 30291 ts_tablet_manager.cc:1397] T 4f130ab719184374966292ad207913fc P 9a25027701d348159e49952160a3e2f2: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260808 22:24:19.948520 30291 raft_consensus.cc:348] T 4f130ab719184374966292ad207913fc P 9a25027701d348159e49952160a3e2f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a25027701d348159e49952160a3e2f2" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 36751 } } peers { permanent_uuid: "5d3016f26e4843dcb074e1dd7f20d17b" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 44513 } } peers { permanent_uuid: "a57a0257a8504929a3d2468f5fd6194c" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 43543 } }
I20260808 22:24:19.948549 30292 raft_consensus.cc:348] T 4f130ab719184374966292ad207913fc P a57a0257a8504929a3d2468f5fd6194c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a25027701d348159e49952160a3e2f2" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 36751 } } peers { permanent_uuid: "5d3016f26e4843dcb074e1dd7f20d17b" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 44513 } } peers { permanent_uuid: "a57a0257a8504929a3d2468f5fd6194c" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 43543 } }
I20260808 22:24:19.948818 30291 raft_consensus.cc:374] T 4f130ab719184374966292ad207913fc P 9a25027701d348159e49952160a3e2f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:24:19.948880 30291 raft_consensus.cc:729] T 4f130ab719184374966292ad207913fc P 9a25027701d348159e49952160a3e2f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a25027701d348159e49952160a3e2f2, State: Initialized, Role: FOLLOWER
I20260808 22:24:19.949028 30292 raft_consensus.cc:374] T 4f130ab719184374966292ad207913fc P a57a0257a8504929a3d2468f5fd6194c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:24:19.949126 30292 raft_consensus.cc:729] T 4f130ab719184374966292ad207913fc P a57a0257a8504929a3d2468f5fd6194c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a57a0257a8504929a3d2468f5fd6194c, State: Initialized, Role: FOLLOWER
I20260808 22:24:19.949482 30291 consensus_queue.cc:260] T 4f130ab719184374966292ad207913fc P 9a25027701d348159e49952160a3e2f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a25027701d348159e49952160a3e2f2" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 36751 } } peers { permanent_uuid: "5d3016f26e4843dcb074e1dd7f20d17b" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 44513 } } peers { permanent_uuid: "a57a0257a8504929a3d2468f5fd6194c" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 43543 } }
I20260808 22:24:19.949708 30292 consensus_queue.cc:260] T 4f130ab719184374966292ad207913fc P a57a0257a8504929a3d2468f5fd6194c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a25027701d348159e49952160a3e2f2" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 36751 } } peers { permanent_uuid: "5d3016f26e4843dcb074e1dd7f20d17b" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 44513 } } peers { permanent_uuid: "a57a0257a8504929a3d2468f5fd6194c" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 43543 } }
I20260808 22:24:19.950632 30296 tablet_bootstrap.cc:492] T 4f130ab719184374966292ad207913fc P 5d3016f26e4843dcb074e1dd7f20d17b: Bootstrap starting.
I20260808 22:24:19.950699 30292 ts_tablet_manager.cc:1428] T 4f130ab719184374966292ad207913fc P a57a0257a8504929a3d2468f5fd6194c: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260808 22:24:19.951134 30261 heartbeater.cc:503] Master 127.9.176.254:42379 was elected leader, sending a full tablet report...
I20260808 22:24:19.953308 30291 ts_tablet_manager.cc:1428] T 4f130ab719184374966292ad207913fc P 9a25027701d348159e49952160a3e2f2: Time spent starting tablet: real 0.009s user 0.004s sys 0.004s
I20260808 22:24:19.953648 29865 heartbeater.cc:503] Master 127.9.176.254:42379 was elected leader, sending a full tablet report...
I20260808 22:24:19.955699 30296 tablet_bootstrap.cc:654] T 4f130ab719184374966292ad207913fc P 5d3016f26e4843dcb074e1dd7f20d17b: Neither blocks nor log segments found. Creating new log.
I20260808 22:24:19.956604 30296 log.cc:824] T 4f130ab719184374966292ad207913fc P 5d3016f26e4843dcb074e1dd7f20d17b: Log is configured to *not* fsync() on all Append() calls
I20260808 22:24:19.959439 30296 tablet_bootstrap.cc:492] T 4f130ab719184374966292ad207913fc P 5d3016f26e4843dcb074e1dd7f20d17b: No bootstrap required, opened a new log
I20260808 22:24:19.959649 30296 ts_tablet_manager.cc:1397] T 4f130ab719184374966292ad207913fc P 5d3016f26e4843dcb074e1dd7f20d17b: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.002s
I20260808 22:24:19.963989 30296 raft_consensus.cc:348] T 4f130ab719184374966292ad207913fc P 5d3016f26e4843dcb074e1dd7f20d17b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a25027701d348159e49952160a3e2f2" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 36751 } } peers { permanent_uuid: "5d3016f26e4843dcb074e1dd7f20d17b" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 44513 } } peers { permanent_uuid: "a57a0257a8504929a3d2468f5fd6194c" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 43543 } }
I20260808 22:24:19.964294 30296 raft_consensus.cc:374] T 4f130ab719184374966292ad207913fc P 5d3016f26e4843dcb074e1dd7f20d17b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:24:19.964357 30296 raft_consensus.cc:729] T 4f130ab719184374966292ad207913fc P 5d3016f26e4843dcb074e1dd7f20d17b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d3016f26e4843dcb074e1dd7f20d17b, State: Initialized, Role: FOLLOWER
I20260808 22:24:19.964846 30296 consensus_queue.cc:260] T 4f130ab719184374966292ad207913fc P 5d3016f26e4843dcb074e1dd7f20d17b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a25027701d348159e49952160a3e2f2" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 36751 } } peers { permanent_uuid: "5d3016f26e4843dcb074e1dd7f20d17b" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 44513 } } peers { permanent_uuid: "a57a0257a8504929a3d2468f5fd6194c" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 43543 } }
I20260808 22:24:19.965936 30296 ts_tablet_manager.cc:1428] T 4f130ab719184374966292ad207913fc P 5d3016f26e4843dcb074e1dd7f20d17b: Time spent starting tablet: real 0.006s user 0.007s sys 0.001s
I20260808 22:24:19.966315 30058 heartbeater.cc:503] Master 127.9.176.254:42379 was elected leader, sending a full tablet report...
Aug 08 22:24:20 dist-test-slave-ncp9 krb5kdc[28123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227859, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.254@KRBTEST.COM for HTTP/127.9.176.212@KRBTEST.COM
W20260808 22:24:20.084105 29866 tablet.cc:2369] T 4f130ab719184374966292ad207913fc P 9a25027701d348159e49952160a3e2f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 22:24:20.092142 30262 tablet.cc:2369] T 4f130ab719184374966292ad207913fc P a57a0257a8504929a3d2468f5fd6194c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 22:24:20.092762 30060 tablet.cc:2369] T 4f130ab719184374966292ad207913fc P 5d3016f26e4843dcb074e1dd7f20d17b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 22:24:20.271550 30298 raft_consensus.cc:482] T 4f130ab719184374966292ad207913fc P 9a25027701d348159e49952160a3e2f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 22:24:20.271816 30298 raft_consensus.cc:504] T 4f130ab719184374966292ad207913fc P 9a25027701d348159e49952160a3e2f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a25027701d348159e49952160a3e2f2" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 36751 } } peers { permanent_uuid: "5d3016f26e4843dcb074e1dd7f20d17b" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 44513 } } peers { permanent_uuid: "a57a0257a8504929a3d2468f5fd6194c" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 43543 } }
I20260808 22:24:20.273370 30298 leader_election.cc:288] T 4f130ab719184374966292ad207913fc P 9a25027701d348159e49952160a3e2f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d3016f26e4843dcb074e1dd7f20d17b (127.9.176.194:44513), a57a0257a8504929a3d2468f5fd6194c (127.9.176.195:43543)
I20260808 22:24:20.278556 30313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:20.273611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.195:43543 (local address 127.9.176.193:43657)
0808 22:24:20.274315 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:20.274327 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 22:24:20.274447 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:24:20.274777 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:24:20.274780 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:24:20.274793 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:24:20.274898 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:24:20.274904 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:24:20.275867 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:24:20.275870 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:24:20.277234 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:24:20.277249 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:24:20.278151 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:24:20.278153 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:24:20.278230 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:20.278247 (+ 17us) client_negotiation.cc:769] Sending connection context
0808 22:24:20.278307 (+ 60us) client_negotiation.cc:240] Negotiation successful
0808 22:24:20.278383 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"mutex_wait_us":16,"thread_start_us":134,"threads_started":1}
I20260808 22:24:20.278543 30290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:20.274030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.193:43657 (local address 127.9.176.195:43543)
0808 22:24:20.274154 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0808 22:24:20.274159 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:24:20.274389 (+ 230us) server_negotiation.cc:407] Connection header received
0808 22:24:20.274555 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:24:20.274558 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:24:20.274610 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:24:20.274654 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:24:20.275013 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:20.275737 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:20.277388 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:20.277944 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:20.278022 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:20.278063 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0808 22:24:20.278359 (+ 296us) server_negotiation.cc:299] Negotiation successful
0808 22:24:20.278436 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 22:24:20.278769 30287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:20.273764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.193:60529 (local address 127.9.176.194:44513)
0808 22:24:20.273897 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0808 22:24:20.273902 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:24:20.274135 (+ 233us) server_negotiation.cc:407] Connection header received
0808 22:24:20.274365 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:24:20.274369 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:24:20.274420 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:24:20.274468 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:24:20.274912 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:20.275719 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:20.277456 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:20.278074 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:20.278169 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:20.278216 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0808 22:24:20.278564 (+ 348us) server_negotiation.cc:299] Negotiation successful
0808 22:24:20.278646 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 22:24:20.279171 30311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:20.273573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.194:44513 (local address 127.9.176.193:60529)
0808 22:24:20.274046 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:20.274059 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 22:24:20.274226 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:24:20.274610 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:24:20.274613 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:24:20.274627 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:24:20.274791 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:24:20.274798 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:24:20.275867 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:24:20.275870 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:24:20.277298 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:24:20.277314 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:24:20.278312 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:24:20.278316 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:24:20.278413 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:20.278435 (+ 22us) client_negotiation.cc:769] Sending connection context
0808 22:24:20.278953 (+ 518us) client_negotiation.cc:240] Negotiation successful
0808 22:24:20.279027 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"mutex_wait_us":56,"thread_start_us":101,"threads_started":1}
I20260808 22:24:20.279649 30190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f130ab719184374966292ad207913fc" candidate_uuid: "9a25027701d348159e49952160a3e2f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a57a0257a8504929a3d2468f5fd6194c" is_pre_election: true
I20260808 22:24:20.279810 29971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f130ab719184374966292ad207913fc" candidate_uuid: "9a25027701d348159e49952160a3e2f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d3016f26e4843dcb074e1dd7f20d17b" is_pre_election: true
I20260808 22:24:20.279955 30190 raft_consensus.cc:2454] T 4f130ab719184374966292ad207913fc P a57a0257a8504929a3d2468f5fd6194c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a25027701d348159e49952160a3e2f2 in term 0.
I20260808 22:24:20.280115 29971 raft_consensus.cc:2454] T 4f130ab719184374966292ad207913fc P 5d3016f26e4843dcb074e1dd7f20d17b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a25027701d348159e49952160a3e2f2 in term 0.
I20260808 22:24:20.280427 29710 leader_election.cc:302] T 4f130ab719184374966292ad207913fc P 9a25027701d348159e49952160a3e2f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a25027701d348159e49952160a3e2f2, a57a0257a8504929a3d2468f5fd6194c; no voters:
I20260808 22:24:20.280705 30298 raft_consensus.cc:2781] T 4f130ab719184374966292ad207913fc P 9a25027701d348159e49952160a3e2f2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 22:24:20.280794 30298 raft_consensus.cc:482] T 4f130ab719184374966292ad207913fc P 9a25027701d348159e49952160a3e2f2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 22:24:20.280838 30298 raft_consensus.cc:3037] T 4f130ab719184374966292ad207913fc P 9a25027701d348159e49952160a3e2f2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:24:20.281894 30298 raft_consensus.cc:504] T 4f130ab719184374966292ad207913fc P 9a25027701d348159e49952160a3e2f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a25027701d348159e49952160a3e2f2" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 36751 } } peers { permanent_uuid: "5d3016f26e4843dcb074e1dd7f20d17b" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 44513 } } peers { permanent_uuid: "a57a0257a8504929a3d2468f5fd6194c" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 43543 } }
I20260808 22:24:20.282245 30298 leader_election.cc:288] T 4f130ab719184374966292ad207913fc P 9a25027701d348159e49952160a3e2f2 [CANDIDATE]: Term 1 election: Requested vote from peers 5d3016f26e4843dcb074e1dd7f20d17b (127.9.176.194:44513), a57a0257a8504929a3d2468f5fd6194c (127.9.176.195:43543)
I20260808 22:24:20.282565 29971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f130ab719184374966292ad207913fc" candidate_uuid: "9a25027701d348159e49952160a3e2f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d3016f26e4843dcb074e1dd7f20d17b"
I20260808 22:24:20.282591 30190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f130ab719184374966292ad207913fc" candidate_uuid: "9a25027701d348159e49952160a3e2f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a57a0257a8504929a3d2468f5fd6194c"
I20260808 22:24:20.282688 29971 raft_consensus.cc:3037] T 4f130ab719184374966292ad207913fc P 5d3016f26e4843dcb074e1dd7f20d17b [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:24:20.282701 30190 raft_consensus.cc:3037] T 4f130ab719184374966292ad207913fc P a57a0257a8504929a3d2468f5fd6194c [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:24:20.283596 30190 raft_consensus.cc:2454] T 4f130ab719184374966292ad207913fc P a57a0257a8504929a3d2468f5fd6194c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a25027701d348159e49952160a3e2f2 in term 1.
I20260808 22:24:20.283887 29971 raft_consensus.cc:2454] T 4f130ab719184374966292ad207913fc P 5d3016f26e4843dcb074e1dd7f20d17b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a25027701d348159e49952160a3e2f2 in term 1.
I20260808 22:24:20.283901 29710 leader_election.cc:302] T 4f130ab719184374966292ad207913fc P 9a25027701d348159e49952160a3e2f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a25027701d348159e49952160a3e2f2, a57a0257a8504929a3d2468f5fd6194c; no voters:
I20260808 22:24:20.284077 30298 raft_consensus.cc:2781] T 4f130ab719184374966292ad207913fc P 9a25027701d348159e49952160a3e2f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:24:20.284297 30298 raft_consensus.cc:686] T 4f130ab719184374966292ad207913fc P 9a25027701d348159e49952160a3e2f2 [term 1 LEADER]: Becoming Leader. State: Replica: 9a25027701d348159e49952160a3e2f2, State: Running, Role: LEADER
I20260808 22:24:20.284621 30298 consensus_queue.cc:237] T 4f130ab719184374966292ad207913fc P 9a25027701d348159e49952160a3e2f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a25027701d348159e49952160a3e2f2" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 36751 } } peers { permanent_uuid: "5d3016f26e4843dcb074e1dd7f20d17b" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 44513 } } peers { permanent_uuid: "a57a0257a8504929a3d2468f5fd6194c" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 43543 } }
I20260808 22:24:20.288167 29578 catalog_manager.cc:5495] T 4f130ab719184374966292ad207913fc P 9a25027701d348159e49952160a3e2f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a25027701d348159e49952160a3e2f2 (127.9.176.193). New cstate: current_term: 1 leader_uuid: "9a25027701d348159e49952160a3e2f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a25027701d348159e49952160a3e2f2" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 36751 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5d3016f26e4843dcb074e1dd7f20d17b" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 44513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a57a0257a8504929a3d2468f5fd6194c" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 43543 } health_report { overall_health: UNKNOWN } } }
I20260808 22:24:20.364769 30284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:20.361061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53142 (local address 127.9.176.193:36751)
0808 22:24:20.361206 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0808 22:24:20.361210 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:24:20.361223 (+ 13us) server_negotiation.cc:407] Connection header received
0808 22:24:20.361303 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:24:20.361305 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:24:20.361358 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:24:20.361449 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:24:20.361704 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:20.362237 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:20.363081 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:20.363299 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:20.363386 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:20.363703 (+ 317us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:24:20.364167 (+ 464us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:24:20.364274 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0808 22:24:20.364557 (+ 283us) server_negotiation.cc:299] Negotiation successful
0808 22:24:20.364624 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 22:24:20.370906 29971 raft_consensus.cc:1261] T 4f130ab719184374966292ad207913fc P 5d3016f26e4843dcb074e1dd7f20d17b [term 1 FOLLOWER]: Refusing update from remote peer 9a25027701d348159e49952160a3e2f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:24:20.370971 30190 raft_consensus.cc:1261] T 4f130ab719184374966292ad207913fc P a57a0257a8504929a3d2468f5fd6194c [term 1 FOLLOWER]: Refusing update from remote peer 9a25027701d348159e49952160a3e2f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:24:20.371892 30319 consensus_queue.cc:1035] T 4f130ab719184374966292ad207913fc P 9a25027701d348159e49952160a3e2f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d3016f26e4843dcb074e1dd7f20d17b" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 44513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:24:20.372080 30298 consensus_queue.cc:1035] T 4f130ab719184374966292ad207913fc P 9a25027701d348159e49952160a3e2f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a57a0257a8504929a3d2468f5fd6194c" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 43543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:24:20.379927 30326 mvcc.cc:205] Tried to move back new op lower bound from 7316389316071612416 to 7316389315728769024. Current Snapshot: MvccSnapshot[applied={T|T < 7316389316071612416}]
I20260808 22:24:20.383529 30330 mvcc.cc:205] Tried to move back new op lower bound from 7316389316071612416 to 7316389315728769024. Current Snapshot: MvccSnapshot[applied={T|T < 7316389316071612416}]
I20260808 22:24:20.390389 29579 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:24:20.390570 29579 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:24:20.393175 29579 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ece099de893c46439733749788a8d611: Sending DeleteTablet for 3 replicas of tablet 4f130ab719184374966292ad207913fc
I20260808 22:24:20.394038 29961 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f130ab719184374966292ad207913fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:24:20 UTC) from {username='kudu', principal='kudu/127.9.176.254@KRBTEST.COM'} at 127.0.0.1:55462
I20260808 22:24:20.394049 29769 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f130ab719184374966292ad207913fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:24:20 UTC) from {username='kudu', principal='kudu/127.9.176.254@KRBTEST.COM'} at 127.0.0.1:53134
I20260808 22:24:20.394551 30145 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f130ab719184374966292ad207913fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:24:20 UTC) from {username='kudu', principal='kudu/127.9.176.254@KRBTEST.COM'} at 127.0.0.1:44226
I20260808 22:24:20.395327 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 29632
I20260808 22:24:20.401598 30340 tablet_replica.cc:330] T 4f130ab719184374966292ad207913fc P 5d3016f26e4843dcb074e1dd7f20d17b: stopping tablet replica
I20260808 22:24:20.402765 30340 raft_consensus.cc:2229] T 4f130ab719184374966292ad207913fc P 5d3016f26e4843dcb074e1dd7f20d17b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:24:20.403267 30340 raft_consensus.cc:2258] T 4f130ab719184374966292ad207913fc P 5d3016f26e4843dcb074e1dd7f20d17b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:24:20.403729 30343 tablet_replica.cc:330] T 4f130ab719184374966292ad207913fc P a57a0257a8504929a3d2468f5fd6194c: stopping tablet replica
I20260808 22:24:20.404042 30343 raft_consensus.cc:2229] T 4f130ab719184374966292ad207913fc P a57a0257a8504929a3d2468f5fd6194c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:24:20.404341 30343 raft_consensus.cc:2258] T 4f130ab719184374966292ad207913fc P a57a0257a8504929a3d2468f5fd6194c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:24:20.406068 30343 ts_tablet_manager.cc:1905] T 4f130ab719184374966292ad207913fc P a57a0257a8504929a3d2468f5fd6194c: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260808 22:24:20.408326 29547 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.176.193:36751 (error 108)
I20260808 22:24:20.408804 30343 ts_tablet_manager.cc:1918] T 4f130ab719184374966292ad207913fc P a57a0257a8504929a3d2468f5fd6194c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 22:24:20.409018 30343 log.cc:1196] T 4f130ab719184374966292ad207913fc P a57a0257a8504929a3d2468f5fd6194c: Deleting WAL directory at /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786227706282457-9923-0/minicluster-data/ts-2/wal/wals/4f130ab719184374966292ad207913fc
I20260808 22:24:20.409416 30343 ts_tablet_manager.cc:1939] T 4f130ab719184374966292ad207913fc P a57a0257a8504929a3d2468f5fd6194c: Deleting consensus metadata
I20260808 22:24:20.409593 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 29867
W20260808 22:24:20.409608 30289 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:24:20.409268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.193:36751 (local address 127.0.0.1:53152)
0808 22:24:20.409406 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:20.409491 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.193:36751: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":39}
W20260808 22:24:20.416350 29547 catalog_manager.cc:4584] TS 9a25027701d348159e49952160a3e2f2 (127.9.176.193:36751): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4f130ab719184374966292ad207913fc: Network error: Client connection negotiation failed: client connection to 127.9.176.193:36751: connect: Connection reset by peer (error 104)
I20260808 22:24:20.417829 29548 catalog_manager.cc:4841] TS a57a0257a8504929a3d2468f5fd6194c (127.9.176.195:43543): tablet 4f130ab719184374966292ad207913fc (table test-table [id=455aedb11db349298f44f9c5328b89be]) successfully deleted
I20260808 22:24:20.422274 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 30062
I20260808 22:24:20.422791 30289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:20.422471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.194:44513 (local address 127.0.0.1:55472)
0808 22:24:20.422648 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:20.422698 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.194:44513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260808 22:24:20.432240 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 29524
I20260808 22:24:20.445564 9923 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260808 22:24:20.779839 9923 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260808 22:24:20.779947 9923 mini_ranger.cc:66] Stopping Ranger...
I20260808 22:24:21.124351 9923 mini_ranger.cc:68] Stopped Ranger
2026-08-08 22:24:21.124 UTC [28163] LOG: received smart shutdown request
2026-08-08 22:24:21.128 UTC [28163] LOG: background worker "logical replication launcher" (PID 28177) exited with exit code 1
2026-08-08 22:24:21.129 UTC [28169] LOG: shutting down
2026-08-08 22:24:21.261 UTC [28163] LOG: database system is shut down
2026-08-08T22:24:21Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (47318 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786227706282457-9923-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 22:24:21 dist-test-slave-ncp9 krb5kdc[30361](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 22:24:21 dist-test-slave-ncp9 krb5kdc[30361](info): set up 2 sockets
Aug 08 22:24:21 dist-test-slave-ncp9 krb5kdc[30361](info): commencing operation
krb5kdc: starting...
W20260808 22:24:22.472277 9923 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786227706282457-9923-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786227706282457-9923-0/krb5kdc/test-user.keytab.
Aug 08 22:24:22 dist-test-slave-ncp9 krb5kdc[30361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227862, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T22:24:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:24:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.176.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.176.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.176.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab.
Entry for principal kudu/127.9.176.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab.
I20260808 22:24:22.587251 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786227706282457-9923-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786227706282457-9923-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786227706282457-9923-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:33445
--webserver_interface=127.9.176.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab
--principal=kudu/127.9.176.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:34785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.176.254:33445,127.9.176.253:34561,127.9.176.252:41631
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:24:22.694746 30377 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:24:22.694985 30377 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:24:22.695022 30377 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:24:22.699147 30377 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:24:22.699209 30377 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:24:22.699227 30377 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:24:22.699242 30377 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:24:22.699255 30377 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:24:22.704419 30377 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:34785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786227706282457-9923-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786227706282457-9923-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.176.254:33445,127.9.176.253:34561,127.9.176.252:41631
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab
--principal=kudu/127.9.176.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:33445
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.176.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30377
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786227706282457-9923-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:24:22.705374 30377 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:24:22.706331 30377 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:24:22.714499 30388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:24:22.714524 30385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:24:22.714646 30377 server_base.cc:1034] running on GCE node
W20260808 22:24:22.714816 30386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:24:22.715325 30377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:24:22.716413 30377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:24:22.717581 30377 hybrid_clock.cc:648] HybridClock initialized: now 1786227862717565 us; error 29 us; skew 500 ppm
Aug 08 22:24:22 dist-test-slave-ncp9 krb5kdc[30361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227862, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:24:22.720223 30377 init.cc:377] Logged in from keytab as kudu/127.9.176.254@KRBTEST.COM (short username kudu)
I20260808 22:24:22.721145 30377 webserver.cc:460] Webserver started at http://127.9.176.254:44771/ using document root <none> and password file <none>
I20260808 22:24:22.721698 30377 fs_manager.cc:359] Metadata directory not provided
I20260808 22:24:22.721765 30377 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:24:22.721987 30377 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:24:22.724035 30377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786227706282457-9923-0/minicluster-data/master-0/data/instance:
uuid: "fd71bc7679e442548dedaff93eb3363b"
format_stamp: "Formatted at 2026-08-08 22:24:22 on dist-test-slave-ncp9"
server_key: "5c195b0ea028a1e596182c43dacab38d"
server_key_iv: "811fcf75b16e8cbb7c9dd0a0ba0e965d"
server_key_version: "encryptionkey@0"
I20260808 22:24:22.724579 30377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786227706282457-9923-0/minicluster-data/master-0/wal/instance:
uuid: "fd71bc7679e442548dedaff93eb3363b"
format_stamp: "Formatted at 2026-08-08 22:24:22 on dist-test-slave-ncp9"
server_key: "5c195b0ea028a1e596182c43dacab38d"
server_key_iv: "811fcf75b16e8cbb7c9dd0a0ba0e965d"
server_key_version: "encryptionkey@0"
I20260808 22:24:22.728091 30377 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260808 22:24:22.730451 30397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:24:22.731563 30377 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 22:24:22.731675 30377 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786227706282457-9923-0/minicluster-data/master-0/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786227706282457-9923-0/minicluster-data/master-0/wal
uuid: "fd71bc7679e442548dedaff93eb3363b"
format_stamp: "Formatted at 2026-08-08 22:24:22 on dist-test-slave-ncp9"
server_key: "5c195b0ea028a1e596182c43dacab38d"
server_key_iv: "811fcf75b16e8cbb7c9dd0a0ba0e965d"
server_key_version: "encryptionkey@0"
I20260808 22:24:22.731761 30377 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786227706282457-9923-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786227706282457-9923-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786227706282457-9923-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:24:22.744324 30377 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:24:22.746850 30377 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:24:22.747023 30377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:24:22.753813 30377 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.254:33445
I20260808 22:24:22.753826 30469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.254:33445 every 8 connection(s)
I20260808 22:24:22.754734 30377 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
I20260808 22:24:22.757459 30470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:24:22.761204 30470 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.176.254" port: 33445 } has no permanent_uuid. Determining permanent_uuid...
I20260808 22:24:22.762512 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 30377
I20260808 22:24:22.762661 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786227706282457-9923-0/minicluster-data/master-0/wal/instance
I20260808 22:24:22.762977 9923 external_mini_cluster.cc:1428] Setting key 763371248a028bcfbc320669f0e099a7
Aug 08 22:24:22 dist-test-slave-ncp9 krb5kdc[30361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227862, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.254@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.176.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.176.253@KRBTEST.COM" created.
I20260808 22:24:22.776067 30473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:22.763547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46406 (local address 127.9.176.254:33445)
0808 22:24:22.763948 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0808 22:24:22.763968 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0808 22:24:22.764216 (+ 248us) server_negotiation.cc:407] Connection header received
0808 22:24:22.765319 (+ 1103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:24:22.765324 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:24:22.765501 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:24:22.765651 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:24:22.766628 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:22.767490 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:22.768507 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:22.768863 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:22.769046 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:22.771341 (+ 2295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:24:22.771347 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:24:22.771360 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:24:22.771387 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:24:22.772862 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:24:22.773401 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:24:22.773406 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:24:22.773407 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:24:22.773509 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:24:22.773921 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:24:22.773924 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:24:22.773926 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:24:22.774306 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:24:22.774426 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0808 22:24:22.774792 (+ 366us) server_negotiation.cc:299] Negotiation successful
0808 22:24:22.775026 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":261,"thread_start_us":117,"threads_started":1}
I20260808 22:24:22.776065 30475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:22.763540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:33445 (local address 127.0.0.1:46406)
0808 22:24:22.764082 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:22.764117 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0808 22:24:22.764991 (+ 874us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:24:22.765697 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:24:22.765703 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:24:22.766153 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:24:22.766477 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:24:22.766495 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:24:22.767612 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:24:22.767615 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:24:22.768396 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:24:22.768406 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:24:22.769019 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:24:22.769023 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:24:22.769131 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:22.769671 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:24:22.769687 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:24:22.771190 (+ 1503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:24:22.772992 (+ 1802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:24:22.773000 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:24:22.773008 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:24:22.773274 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:24:22.773643 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:24:22.773647 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:24:22.773648 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:24:22.773816 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:24:22.774418 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:24:22.774424 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:24:22.774567 (+ 143us) client_negotiation.cc:769] Sending connection context
0808 22:24:22.774714 (+ 147us) client_negotiation.cc:240] Negotiation successful
0808 22:24:22.775087 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"mutex_wait_us":93,"thread_start_us":145,"threads_started":1}
I20260808 22:24:22.778396 30470 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.176.253" port: 34561 } has no permanent_uuid. Determining permanent_uuid...
I20260808 22:24:22.779110 30475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:22.778872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.253:34561 (local address 127.0.0.1:58012)
0808 22:24:22.778981 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:22.779051 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.253:34561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260808 22:24:22.779323 30401 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.176.253:34561: connect: Connection refused (error 111)
I20260808 22:24:22.780157 30475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:22.779980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.253:34561 (local address 127.0.0.1:58020)
0808 22:24:22.780067 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:22.780108 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.253:34561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260808 22:24:22.780295 30470 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.176.253:34561: Network error: Client connection negotiation failed: client connection to 127.9.176.253:34561: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.176.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.253.keytab.
Entry for principal kudu/127.9.176.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.253.keytab.
I20260808 22:24:22.787364 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786227706282457-9923-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786227706282457-9923-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786227706282457-9923-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786227706282457-9923-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.253:34561
--webserver_interface=127.9.176.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.253.keytab
--principal=kudu/127.9.176.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:34785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.176.254:33445,127.9.176.253:34561,127.9.176.252:41631
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 22:24:22.829520 30470 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.176.253" port: 34561 } attempt: 1
I20260808 22:24:22.830377 30475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:22.830090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.253:34561 (local address 127.0.0.1:58024)
0808 22:24:22.830240 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:22.830299 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.253:34561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260808 22:24:22.831037 30475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:22.830853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.253:34561 (local address 127.0.0.1:58040)
0808 22:24:22.830938 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:22.830981 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.253:34561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260808 22:24:22.831154 30470 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.176.253:34561: Network error: Client connection negotiation failed: client connection to 127.9.176.253:34561: connect: Connection refused (error 111)
W20260808 22:24:22.897090 30479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:24:22.897352 30479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:24:22.897392 30479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260808 22:24:22.899307 30470 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.176.253" port: 34561 } attempt: 2
I20260808 22:24:22.900122 30475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:22.899838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.253:34561 (local address 127.0.0.1:58042)
0808 22:24:22.899982 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:22.900045 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.253:34561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260808 22:24:22.900727 30475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:22.900548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.253:34561 (local address 127.0.0.1:58058)
0808 22:24:22.900629 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:22.900672 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.253:34561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260808 22:24:22.900840 30470 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.176.253:34561: Network error: Client connection negotiation failed: client connection to 127.9.176.253:34561: connect: Connection refused (error 111)
W20260808 22:24:22.901621 30479 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:24:22.901675 30479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:24:22.901696 30479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:24:22.901726 30479 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:24:22.901757 30479 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:24:22.906975 30479 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:34785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786227706282457-9923-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786227706282457-9923-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.9.176.254:33445,127.9.176.253:34561,127.9.176.252:41631
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.253.keytab
--principal=kudu/127.9.176.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.253:34561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786227706282457-9923-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.9.176.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786227706282457-9923-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:24:22.907974 30479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:24:22.909004 30479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:24:22.917249 30488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:24:22.917249 30491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:24:22.917372 30479 server_base.cc:1034] running on GCE node
W20260808 22:24:22.917570 30486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:24:22.918095 30479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:24:22.919100 30479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:24:22.920253 30479 hybrid_clock.cc:648] HybridClock initialized: now 1786227862920226 us; error 37 us; skew 500 ppm
Aug 08 22:24:22 dist-test-slave-ncp9 krb5kdc[30361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227862, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:24:22.922984 30479 init.cc:377] Logged in from keytab as kudu/127.9.176.253@KRBTEST.COM (short username kudu)
I20260808 22:24:22.923800 30479 webserver.cc:460] Webserver started at http://127.9.176.253:36743/ using document root <none> and password file <none>
I20260808 22:24:22.924288 30479 fs_manager.cc:359] Metadata directory not provided
I20260808 22:24:22.924343 30479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:24:22.924507 30479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:24:22.926512 30479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786227706282457-9923-0/minicluster-data/master-1/data/instance:
uuid: "e88c02461e15448f966d535d8ea2f340"
format_stamp: "Formatted at 2026-08-08 22:24:22 on dist-test-slave-ncp9"
server_key: "938dd05c54350b424d3d2e763560df9d"
server_key_iv: "7046c7f2fa758e1b02402521c0f9de1a"
server_key_version: "encryptionkey@0"
I20260808 22:24:22.927027 30479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786227706282457-9923-0/minicluster-data/master-1/wal/instance:
uuid: "e88c02461e15448f966d535d8ea2f340"
format_stamp: "Formatted at 2026-08-08 22:24:22 on dist-test-slave-ncp9"
server_key: "938dd05c54350b424d3d2e763560df9d"
server_key_iv: "7046c7f2fa758e1b02402521c0f9de1a"
server_key_version: "encryptionkey@0"
I20260808 22:24:22.930555 30479 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 22:24:22.932771 30500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:24:22.933938 30479 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 22:24:22.934039 30479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786227706282457-9923-0/minicluster-data/master-1/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786227706282457-9923-0/minicluster-data/master-1/wal
uuid: "e88c02461e15448f966d535d8ea2f340"
format_stamp: "Formatted at 2026-08-08 22:24:22 on dist-test-slave-ncp9"
server_key: "938dd05c54350b424d3d2e763560df9d"
server_key_iv: "7046c7f2fa758e1b02402521c0f9de1a"
server_key_version: "encryptionkey@0"
I20260808 22:24:22.934124 30479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786227706282457-9923-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786227706282457-9923-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786227706282457-9923-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:24:22.964576 30479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:24:22.967815 30479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:24:22.967976 30479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:24:22.974990 30479 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.253:34561
I20260808 22:24:22.975008 30577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.253:34561 every 8 connection(s)
I20260808 22:24:22.975934 30479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786227706282457-9923-0/minicluster-data/master-1/data/info.pb
I20260808 22:24:22.978677 30578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:24:22.982424 30578 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.176.254" port: 33445 } has no permanent_uuid. Determining permanent_uuid...
I20260808 22:24:22.983435 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 30479
I20260808 22:24:22.983565 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786227706282457-9923-0/minicluster-data/master-1/wal/instance
I20260808 22:24:22.983860 9923 external_mini_cluster.cc:1428] Setting key b9a7fa767e1f21686717045c1f4af5b7
Aug 08 22:24:22 dist-test-slave-ncp9 krb5kdc[30361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227862, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.253@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.176.252@KRBTEST.COM; defaulting to no policy
I20260808 22:24:22.992000 30470 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.176.253" port: 34561 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.176.252@KRBTEST.COM" created.
I20260808 22:24:22.996037 30473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:22.984424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46414 (local address 127.9.176.254:33445)
0808 22:24:22.984588 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0808 22:24:22.984593 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:24:22.985416 (+ 823us) server_negotiation.cc:407] Connection header received
0808 22:24:22.986340 (+ 924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:24:22.986344 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:24:22.986402 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:24:22.986472 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:24:22.987609 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:22.988189 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:22.989359 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:22.989596 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:22.989686 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:22.992118 (+ 2432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:24:22.992121 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:24:22.992124 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:24:22.992151 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:24:22.993750 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:24:22.994361 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:24:22.994365 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:24:22.994367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:24:22.994459 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:24:22.994841 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:24:22.994844 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:24:22.994846 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:24:22.995068 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:24:22.995164 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0808 22:24:22.995745 (+ 581us) server_negotiation.cc:299] Negotiation successful
0808 22:24:22.995874 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 22:24:22.997567 30581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:22.984781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:33445 (local address 127.0.0.1:46414)
0808 22:24:22.985256 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:22.985309 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0808 22:24:22.986137 (+ 828us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:24:22.986654 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:24:22.986661 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:24:22.986986 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:24:22.987460 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:24:22.987473 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:24:22.988314 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:24:22.988317 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:24:22.989239 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:24:22.989249 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:24:22.989736 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:24:22.989740 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:24:22.989903 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:22.990436 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:24:22.990452 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:24:22.991946 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:24:22.993903 (+ 1957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:24:22.993910 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:24:22.993925 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:24:22.994273 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:24:22.994554 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:24:22.994557 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:24:22.994559 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:24:22.994739 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:24:22.995166 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:24:22.995173 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:24:22.995463 (+ 290us) client_negotiation.cc:769] Sending connection context
0808 22:24:22.996265 (+ 802us) client_negotiation.cc:240] Negotiation successful
0808 22:24:22.996618 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":136,"threads_started":1}
Aug 08 22:24:22 dist-test-slave-ncp9 krb5kdc[30361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227862, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.254@KRBTEST.COM for kudu/127.9.176.253@KRBTEST.COM
I20260808 22:24:22.998932 30578 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.176.253" port: 34561 } has no permanent_uuid. Determining permanent_uuid...
I20260808 22:24:23.002558 30475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:22.992410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.253:34561 (local address 127.0.0.1:58066)
0808 22:24:22.992523 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:22.992538 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 22:24:22.992614 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:24:22.993410 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:24:22.993414 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:24:22.993656 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:24:22.994227 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:24:22.994239 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:24:22.995392 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:24:22.995395 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:24:22.996058 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:24:22.996068 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:24:22.996712 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:24:22.996716 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:24:22.996822 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:22.997268 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:24:22.997282 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:24:22.998743 (+ 1461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:24:23.000860 (+ 2117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:24:23.000863 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:24:23.000865 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:24:23.001128 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:24:23.001428 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:24:23.001431 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:24:23.001433 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:24:23.001525 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:24:23.002017 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:24:23.002020 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:24:23.002123 (+ 103us) client_negotiation.cc:769] Sending connection context
0808 22:24:23.002253 (+ 130us) client_negotiation.cc:240] Negotiation successful
0808 22:24:23.002387 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260808 22:24:23.002991 30586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:22.992659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58066 (local address 127.9.176.253:34561)
0808 22:24:22.992980 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0808 22:24:22.992990 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 22:24:22.993011 (+ 21us) server_negotiation.cc:407] Connection header received
0808 22:24:22.993050 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:24:22.993056 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:24:22.993257 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:24:22.993400 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:24:22.994366 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:22.995285 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:22.996199 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:22.996601 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:22.996698 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:22.998906 (+ 2208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:24:22.998917 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:24:22.998931 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:24:22.998957 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:24:23.000769 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:24:23.001239 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:24:23.001245 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:24:23.001247 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:24:23.001336 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:24:23.001614 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:24:23.001617 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:24:23.001619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:24:23.001902 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:24:23.002044 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0808 22:24:23.002359 (+ 315us) server_negotiation.cc:299] Negotiation successful
0808 22:24:23.002535 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":199,"thread_start_us":87,"threads_started":1}
I20260808 22:24:23.005015 30470 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.176.252" port: 41631 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 22:24:23 dist-test-slave-ncp9 krb5kdc[30361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227862, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.253@KRBTEST.COM for kudu/127.9.176.253@KRBTEST.COM
I20260808 22:24:23.005713 30475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:23.005461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.252:41631 (local address 127.0.0.1:38050)
0808 22:24:23.005589 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:23.005644 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.252:41631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260808 22:24:23.006448 30475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:23.006175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.252:41631 (local address 127.0.0.1:38060)
0808 22:24:23.006331 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:23.006384 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.252:41631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
W20260808 22:24:23.006589 30470 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.176.252:41631: Network error: Client connection negotiation failed: client connection to 127.9.176.252:41631: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.176.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.252.keytab.
Entry for principal kudu/127.9.176.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.252.keytab.
I20260808 22:24:23.009248 30581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:22.999394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.253:34561 (local address 127.0.0.1:58080)
0808 22:24:22.999510 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:22.999524 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 22:24:22.999599 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:24:23.000344 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:24:23.000347 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:24:23.000555 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:24:23.000669 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:24:23.000676 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:24:23.001551 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:24:23.001554 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:24:23.002279 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:24:23.002289 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:24:23.002872 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:24:23.002875 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:24:23.002962 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:23.003380 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:24:23.003393 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:24:23.005949 (+ 2556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:24:23.007683 (+ 1734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:24:23.007686 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:24:23.007689 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:24:23.007937 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:24:23.008205 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:24:23.008208 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:24:23.008210 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:24:23.008308 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:24:23.008694 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:24:23.008697 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:24:23.008779 (+ 82us) client_negotiation.cc:769] Sending connection context
0808 22:24:23.008861 (+ 82us) client_negotiation.cc:240] Negotiation successful
0808 22:24:23.009055 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260808 22:24:23.009298 30589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:22.999600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58080 (local address 127.9.176.253:34561)
0808 22:24:23.000111 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0808 22:24:23.000121 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 22:24:23.000134 (+ 13us) server_negotiation.cc:407] Connection header received
0808 22:24:23.000179 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:24:23.000182 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:24:23.000225 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:24:23.000510 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:24:23.000784 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:23.001473 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:23.002392 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:23.002757 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:23.003152 (+ 395us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:23.006126 (+ 2974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:24:23.006130 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:24:23.006133 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:24:23.006158 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:24:23.007581 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:24:23.008024 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:24:23.008027 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:24:23.008029 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:24:23.008107 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:24:23.008389 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:24:23.008392 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:24:23.008394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:24:23.008588 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:24:23.008674 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0808 22:24:23.008937 (+ 263us) server_negotiation.cc:299] Negotiation successful
0808 22:24:23.009127 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":86,"server-negotiator.queue_time_us":442,"thread_start_us":159,"threads_started":1}
I20260808 22:24:23.010049 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786227706282457-9923-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786227706282457-9923-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786227706282457-9923-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786227706282457-9923-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.252:41631
--webserver_interface=127.9.176.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.252.keytab
--principal=kudu/127.9.176.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:34785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.176.254:33445,127.9.176.253:34561,127.9.176.252:41631
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 22:24:23.010123 30578 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.176.252" port: 41631 } has no permanent_uuid. Determining permanent_uuid...
I20260808 22:24:23.010802 30581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:23.010522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.252:41631 (local address 127.0.0.1:38074)
0808 22:24:23.010643 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:23.010738 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.252:41631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260808 22:24:23.010995 30505 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.176.252:41631: connect: Connection refused (error 111)
I20260808 22:24:23.011827 30581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:23.011626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.252:41631 (local address 127.0.0.1:38088)
0808 22:24:23.011727 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:23.011776 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.252:41631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260808 22:24:23.011952 30578 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.176.252:41631: Network error: Client connection negotiation failed: client connection to 127.9.176.252:41631: connect: Connection refused (error 111)
I20260808 22:24:23.037740 30470 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.176.252" port: 41631 } attempt: 1
I20260808 22:24:23.038553 30475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:23.038261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.252:41631 (local address 127.0.0.1:38102)
0808 22:24:23.038396 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:23.038457 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.252:41631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260808 22:24:23.039237 30475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:23.039037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.252:41631 (local address 127.0.0.1:38106)
0808 22:24:23.039138 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:23.039182 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.252:41631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260808 22:24:23.039343 30470 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.176.252:41631: Network error: Client connection negotiation failed: client connection to 127.9.176.252:41631: connect: Connection refused (error 111)
I20260808 22:24:23.061228 30578 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.176.252" port: 41631 } attempt: 1
I20260808 22:24:23.062156 30581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:23.061857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.252:41631 (local address 127.0.0.1:38120)
0808 22:24:23.062014 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:23.062078 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.252:41631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260808 22:24:23.062860 30581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:23.062653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.252:41631 (local address 127.0.0.1:38132)
0808 22:24:23.062756 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:23.062802 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.252:41631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260808 22:24:23.062973 30578 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.176.252:41631: Network error: Client connection negotiation failed: client connection to 127.9.176.252:41631: connect: Connection refused (error 111)
I20260808 22:24:23.114494 30470 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.176.252" port: 41631 } attempt: 2
I20260808 22:24:23.115288 30475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:23.115005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.252:41631 (local address 127.0.0.1:38148)
0808 22:24:23.115147 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:23.115207 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.252:41631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260808 22:24:23.116044 30475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:23.115848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.252:41631 (local address 127.0.0.1:38160)
0808 22:24:23.115945 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:23.115992 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.252:41631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260808 22:24:23.116165 30470 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.176.252:41631: Network error: Client connection negotiation failed: client connection to 127.9.176.252:41631: connect: Connection refused (error 111)
W20260808 22:24:23.125942 30593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:24:23.126197 30593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:24:23.126266 30593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:24:23.130529 30593 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:24:23.130594 30593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:24:23.130616 30593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:24:23.130652 30593 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:24:23.130684 30593 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:24:23.131114 30578 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.176.252" port: 41631 } attempt: 2
I20260808 22:24:23.131845 30581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:23.131571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.252:41631 (local address 127.0.0.1:38176)
0808 22:24:23.131713 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:23.131773 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.252:41631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260808 22:24:23.132567 30581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:23.132382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.252:41631 (local address 127.0.0.1:38186)
0808 22:24:23.132467 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:23.132511 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.252:41631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260808 22:24:23.132679 30578 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.176.252:41631: Network error: Client connection negotiation failed: client connection to 127.9.176.252:41631: connect: Connection refused (error 111)
I20260808 22:24:23.136303 30593 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:34785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786227706282457-9923-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786227706282457-9923-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.9.176.254:33445,127.9.176.253:34561,127.9.176.252:41631
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.252.keytab
--principal=kudu/127.9.176.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.252:41631
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786227706282457-9923-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.9.176.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30593
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786227706282457-9923-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:24:23.137446 30593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:24:23.138477 30593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 22:24:23.146768 30593 server_base.cc:1034] running on GCE node
W20260808 22:24:23.146718 30601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:24:23.146718 30603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:24:23.146713 30600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:24:23.147532 30593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:24:23.148581 30593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:24:23.149739 30593 hybrid_clock.cc:648] HybridClock initialized: now 1786227863149713 us; error 38 us; skew 500 ppm
Aug 08 22:24:23 dist-test-slave-ncp9 krb5kdc[30361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227863, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:24:23.152460 30593 init.cc:377] Logged in from keytab as kudu/127.9.176.252@KRBTEST.COM (short username kudu)
I20260808 22:24:23.153383 30593 webserver.cc:460] Webserver started at http://127.9.176.252:37573/ using document root <none> and password file <none>
I20260808 22:24:23.153887 30593 fs_manager.cc:359] Metadata directory not provided
I20260808 22:24:23.153960 30593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:24:23.154139 30593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:24:23.156152 30593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786227706282457-9923-0/minicluster-data/master-2/data/instance:
uuid: "86db710979c54485a00d238bccfc957e"
format_stamp: "Formatted at 2026-08-08 22:24:23 on dist-test-slave-ncp9"
server_key: "a7b3ac90a549a0afbbd7d981cf441393"
server_key_iv: "b6d298ce01d2acdd700b0e1d51e1ab39"
server_key_version: "encryptionkey@0"
I20260808 22:24:23.156687 30593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786227706282457-9923-0/minicluster-data/master-2/wal/instance:
uuid: "86db710979c54485a00d238bccfc957e"
format_stamp: "Formatted at 2026-08-08 22:24:23 on dist-test-slave-ncp9"
server_key: "a7b3ac90a549a0afbbd7d981cf441393"
server_key_iv: "b6d298ce01d2acdd700b0e1d51e1ab39"
server_key_version: "encryptionkey@0"
I20260808 22:24:23.160310 30593 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260808 22:24:23.162755 30616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:24:23.163905 30593 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260808 22:24:23.164006 30593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786227706282457-9923-0/minicluster-data/master-2/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786227706282457-9923-0/minicluster-data/master-2/wal
uuid: "86db710979c54485a00d238bccfc957e"
format_stamp: "Formatted at 2026-08-08 22:24:23 on dist-test-slave-ncp9"
server_key: "a7b3ac90a549a0afbbd7d981cf441393"
server_key_iv: "b6d298ce01d2acdd700b0e1d51e1ab39"
server_key_version: "encryptionkey@0"
I20260808 22:24:23.164088 30593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786227706282457-9923-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786227706282457-9923-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786227706282457-9923-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:24:23.182261 30593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:24:23.185074 30593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:24:23.185236 30593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:24:23.191941 30593 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.252:41631
I20260808 22:24:23.191963 30688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.252:41631 every 8 connection(s)
I20260808 22:24:23.192867 30593 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786227706282457-9923-0/minicluster-data/master-2/data/info.pb
I20260808 22:24:23.195554 30689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:24:23.197106 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 30593
I20260808 22:24:23.197233 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786227706282457-9923-0/minicluster-data/master-2/wal/instance
I20260808 22:24:23.197520 9923 external_mini_cluster.cc:1428] Setting key 8d9986ba8f638a8591fdf3abe56e39b9
I20260808 22:24:23.199831 30689 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.176.254" port: 33445 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 22:24:23 dist-test-slave-ncp9 krb5kdc[30361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227863, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.252@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
Aug 08 22:24:23 dist-test-slave-ncp9 krb5kdc[30361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227862, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.176.252@KRBTEST.COM
I20260808 22:24:23.210407 30473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:23.201756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46416 (local address 127.9.176.254:33445)
0808 22:24:23.201875 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0808 22:24:23.201878 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:24:23.202125 (+ 247us) server_negotiation.cc:407] Connection header received
0808 22:24:23.202262 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:24:23.202266 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:24:23.202316 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:24:23.202392 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:24:23.203061 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:23.203543 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:23.204410 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:23.204637 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:23.204734 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:23.206950 (+ 2216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:24:23.206953 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:24:23.206956 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:24:23.206984 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:24:23.208476 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:24:23.208985 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:24:23.208989 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:24:23.208991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:24:23.209064 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:24:23.209371 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:24:23.209374 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:24:23.209376 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:24:23.209561 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:24:23.209646 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0808 22:24:23.210123 (+ 477us) server_negotiation.cc:299] Negotiation successful
0808 22:24:23.210247 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260808 22:24:23.211243 30699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:23.201647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:33445 (local address 127.0.0.1:46416)
0808 22:24:23.201988 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:23.202038 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0808 22:24:23.202159 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:24:23.202453 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:24:23.202457 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:24:23.202794 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:24:23.202924 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:24:23.202931 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:24:23.203658 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:24:23.203660 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:24:23.204297 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:24:23.204306 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:24:23.204779 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:24:23.204783 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:24:23.204884 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:23.205343 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:24:23.205363 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:24:23.206804 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:24:23.208575 (+ 1771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:24:23.208584 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:24:23.208593 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:24:23.208872 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:24:23.209147 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:24:23.209150 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:24:23.209152 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:24:23.209283 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:24:23.209673 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:24:23.209679 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:24:23.209904 (+ 225us) client_negotiation.cc:769] Sending connection context
0808 22:24:23.210064 (+ 160us) client_negotiation.cc:240] Negotiation successful
0808 22:24:23.210356 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":91,"threads_started":1}
I20260808 22:24:23.211261 30693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:23.198937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38196 (local address 127.9.176.252:41631)
0808 22:24:23.199371 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0808 22:24:23.199388 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 22:24:23.199422 (+ 34us) server_negotiation.cc:407] Connection header received
0808 22:24:23.200070 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:24:23.200087 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:24:23.200432 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:24:23.200750 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:24:23.201573 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:23.202915 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:23.203742 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:23.204204 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:23.204344 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:23.207385 (+ 3041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:24:23.207390 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:24:23.207399 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:24:23.207419 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:24:23.209039 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:24:23.209485 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:24:23.209492 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:24:23.209494 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:24:23.209580 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:24:23.209870 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:24:23.209874 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:24:23.209876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:24:23.210123 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:24:23.210321 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0808 22:24:23.210555 (+ 234us) server_negotiation.cc:299] Negotiation successful
0808 22:24:23.210691 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":128,"threads_started":1}
I20260808 22:24:23.212684 30689 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.176.253" port: 34561 } has no permanent_uuid. Determining permanent_uuid...
I20260808 22:24:23.215310 30470 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.176.252" port: 41631 } attempt: 3
Aug 08 22:24:23 dist-test-slave-ncp9 krb5kdc[30361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227862, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
Aug 08 22:24:23 dist-test-slave-ncp9 krb5kdc[30361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227863, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.252@KRBTEST.COM for kudu/127.9.176.253@KRBTEST.COM
Aug 08 22:24:23 dist-test-slave-ncp9 krb5kdc[30361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227862, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.254@KRBTEST.COM for kudu/127.9.176.252@KRBTEST.COM
I20260808 22:24:23.222721 30473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:23.213525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46428 (local address 127.9.176.254:33445)
0808 22:24:23.213632 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0808 22:24:23.213635 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:24:23.213648 (+ 13us) server_negotiation.cc:407] Connection header received
0808 22:24:23.213757 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:24:23.213759 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:24:23.213794 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:24:23.213860 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:24:23.214510 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:23.215207 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:23.216174 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:23.216466 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:23.216555 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:23.218891 (+ 2336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:24:23.218894 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:24:23.218897 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:24:23.218918 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:24:23.220598 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:24:23.221056 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:24:23.221060 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:24:23.221061 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:24:23.221142 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:24:23.221395 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:24:23.221398 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:24:23.221399 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:24:23.221607 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:24:23.221831 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0808 22:24:23.222445 (+ 614us) server_negotiation.cc:299] Negotiation successful
0808 22:24:23.222556 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260808 22:24:23.222728 30589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:23.213313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58082 (local address 127.9.176.253:34561)
0808 22:24:23.213432 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0808 22:24:23.213437 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:24:23.213480 (+ 43us) server_negotiation.cc:407] Connection header received
0808 22:24:23.213630 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:24:23.213633 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:24:23.213692 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:24:23.213744 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:24:23.214302 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:23.215074 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:23.216172 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:23.216453 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:23.216530 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:23.219401 (+ 2871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:24:23.219405 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:24:23.219407 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:24:23.219433 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:24:23.220957 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:24:23.221426 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:24:23.221429 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:24:23.221431 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:24:23.221509 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:24:23.221829 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:24:23.221832 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:24:23.221834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:24:23.222041 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:24:23.222106 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0808 22:24:23.222472 (+ 366us) server_negotiation.cc:299] Negotiation successful
0808 22:24:23.222580 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260808 22:24:23.223134 30699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:23.213183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.253:34561 (local address 127.0.0.1:58082)
0808 22:24:23.213427 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:23.213440 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 22:24:23.213517 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:24:23.213886 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:24:23.213889 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:24:23.214046 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:24:23.214160 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:24:23.214166 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:24:23.215182 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:24:23.215184 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:24:23.216026 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:24:23.216037 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:24:23.216702 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:24:23.216706 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:24:23.217105 (+ 399us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:23.217549 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:24:23.217564 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:24:23.219235 (+ 1671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:24:23.221083 (+ 1848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:24:23.221086 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:24:23.221088 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:24:23.221303 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:24:23.221618 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:24:23.221621 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:24:23.221623 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:24:23.221723 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:24:23.222193 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:24:23.222196 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:24:23.222288 (+ 92us) client_negotiation.cc:769] Sending connection context
0808 22:24:23.222873 (+ 585us) client_negotiation.cc:240] Negotiation successful
0808 22:24:23.223000 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185}
I20260808 22:24:23.223809 30578 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.176.252" port: 41631 } attempt: 3
I20260808 22:24:23.223922 30689 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.176.252" port: 41631 } has no permanent_uuid. Determining permanent_uuid...
I20260808 22:24:23.224031 30475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:23.215638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.252:41631 (local address 127.0.0.1:38200)
0808 22:24:23.215788 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:23.215800 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 22:24:23.215875 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:24:23.216075 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:24:23.216078 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:24:23.216241 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:24:23.216318 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:24:23.216323 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:24:23.217630 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:24:23.217633 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:24:23.218305 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:24:23.218316 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:24:23.218764 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:24:23.218768 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:24:23.218843 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:23.219255 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:24:23.219270 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:24:23.221086 (+ 1816us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:24:23.222651 (+ 1565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:24:23.222655 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:24:23.222657 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:24:23.222853 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:24:23.223101 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:24:23.223104 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:24:23.223106 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:24:23.223193 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:24:23.223617 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:24:23.223621 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:24:23.223707 (+ 86us) client_negotiation.cc:769] Sending connection context
0808 22:24:23.223782 (+ 75us) client_negotiation.cc:240] Negotiation successful
0808 22:24:23.223900 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93}
I20260808 22:24:23.224696 30693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:23.215703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38200 (local address 127.9.176.252:41631)
0808 22:24:23.215789 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0808 22:24:23.215793 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:24:23.215840 (+ 47us) server_negotiation.cc:407] Connection header received
0808 22:24:23.215948 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:24:23.215951 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:24:23.215985 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:24:23.216044 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:24:23.216427 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:23.217499 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:23.218432 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:23.218662 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:23.218747 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:23.221217 (+ 2470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:24:23.221221 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:24:23.221224 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:24:23.221247 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:24:23.222560 (+ 1313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:24:23.222939 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:24:23.222943 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:24:23.222944 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:24:23.223015 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:24:23.223292 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:24:23.223295 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:24:23.223296 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:24:23.223493 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:24:23.224363 (+ 870us) server_negotiation.cc:1035] Waiting for connection context
0808 22:24:23.224453 (+ 90us) server_negotiation.cc:299] Negotiation successful
0808 22:24:23.224572 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260808 22:24:23.228330 30470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd71bc7679e442548dedaff93eb3363b: Bootstrap starting.
Aug 08 22:24:23 dist-test-slave-ncp9 krb5kdc[30361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227863, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.252@KRBTEST.COM for kudu/127.9.176.252@KRBTEST.COM
Aug 08 22:24:23 dist-test-slave-ncp9 krb5kdc[30361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227862, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.253@KRBTEST.COM for kudu/127.9.176.252@KRBTEST.COM
Aug 08 22:24:23 dist-test-slave-ncp9 krb5kdc[30361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227862, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.176.253@KRBTEST.COM
I20260808 22:24:23.231389 30470 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd71bc7679e442548dedaff93eb3363b: Neither blocks nor log segments found. Creating new log.
I20260808 22:24:23.232329 30470 log.cc:824] T 00000000000000000000000000000000 P fd71bc7679e442548dedaff93eb3363b: Log is configured to *not* fsync() on all Append() calls
I20260808 22:24:23.233304 30699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:23.224287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.252:41631 (local address 127.0.0.1:38222)
0808 22:24:23.224434 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:23.224447 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 22:24:23.224512 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:24:23.225199 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:24:23.225202 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:24:23.225319 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:24:23.225375 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:24:23.225379 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:24:23.226290 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:24:23.226292 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:24:23.226983 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:24:23.226994 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:24:23.227616 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:24:23.227619 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:24:23.227704 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:23.228098 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:24:23.228110 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:24:23.229939 (+ 1829us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:24:23.231827 (+ 1888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:24:23.231830 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:24:23.231832 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:24:23.232041 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:24:23.232302 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:24:23.232304 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:24:23.232306 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:24:23.232386 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:24:23.232809 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:24:23.232812 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:24:23.232932 (+ 120us) client_negotiation.cc:769] Sending connection context
0808 22:24:23.233034 (+ 102us) client_negotiation.cc:240] Negotiation successful
0808 22:24:23.233158 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90}
I20260808 22:24:23.233881 30700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:23.224797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38222 (local address 127.9.176.252:41631)
0808 22:24:23.224911 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0808 22:24:23.224920 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 22:24:23.224932 (+ 12us) server_negotiation.cc:407] Connection header received
0808 22:24:23.224965 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:24:23.224968 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:24:23.225006 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:24:23.225083 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:24:23.225472 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:23.226177 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:23.227098 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:23.227431 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:23.227524 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:23.230066 (+ 2542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:24:23.230069 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:24:23.230071 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:24:23.230092 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:24:23.231719 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:24:23.232136 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:24:23.232138 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:24:23.232140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:24:23.232220 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:24:23.232472 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:24:23.232474 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:24:23.232476 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:24:23.232685 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:24:23.232775 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0808 22:24:23.233633 (+ 858us) server_negotiation.cc:299] Negotiation successful
0808 22:24:23.233756 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260808 22:24:23.234458 30581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:23.224205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.252:41631 (local address 127.0.0.1:38210)
0808 22:24:23.224434 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:23.224447 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 22:24:23.224511 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:24:23.225072 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:24:23.225075 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:24:23.225226 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:24:23.225322 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:24:23.225328 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:24:23.226316 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:24:23.226320 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:24:23.227156 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:24:23.227165 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:24:23.227710 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:24:23.227714 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:24:23.227800 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:23.228433 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:24:23.228447 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:24:23.229955 (+ 1508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:24:23.232951 (+ 2996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:24:23.232954 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:24:23.232956 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:24:23.233215 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:24:23.233500 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:24:23.233503 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:24:23.233504 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:24:23.233606 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:24:23.234010 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:24:23.234013 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:24:23.234103 (+ 90us) client_negotiation.cc:769] Sending connection context
0808 22:24:23.234181 (+ 78us) client_negotiation.cc:240] Negotiation successful
0808 22:24:23.234303 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"mutex_wait_us":5}
I20260808 22:24:23.234827 30470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd71bc7679e442548dedaff93eb3363b: No bootstrap required, opened a new log
I20260808 22:24:23.235141 30693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:23.224582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38210 (local address 127.9.176.252:41631)
0808 22:24:23.224830 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0808 22:24:23.224833 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:24:23.224845 (+ 12us) server_negotiation.cc:407] Connection header received
0808 22:24:23.224877 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:24:23.224880 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:24:23.224947 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:24:23.225426 (+ 479us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:24:23.225508 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:23.226187 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:23.227286 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:23.227585 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:23.229052 (+ 1467us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:23.230274 (+ 1222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:24:23.230277 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:24:23.230280 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:24:23.230304 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:24:23.232787 (+ 2483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:24:23.233317 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:24:23.233320 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:24:23.233322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:24:23.233393 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:24:23.233716 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:24:23.233719 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:24:23.233721 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:24:23.233912 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:24:23.234829 (+ 917us) server_negotiation.cc:1035] Waiting for connection context
0808 22:24:23.234901 (+ 72us) server_negotiation.cc:299] Negotiation successful
0808 22:24:23.235003 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":129,"threads_started":1}
I20260808 22:24:23.235162 30589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:23.224887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58096 (local address 127.9.176.253:34561)
0808 22:24:23.225011 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0808 22:24:23.225015 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:24:23.225635 (+ 620us) server_negotiation.cc:407] Connection header received
0808 22:24:23.225789 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:24:23.225792 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:24:23.225829 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:24:23.225872 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:24:23.226627 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:23.227310 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:23.228290 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:23.228623 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:23.228695 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:23.231592 (+ 2897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:24:23.231596 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:24:23.231598 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:24:23.231623 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:24:23.233168 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:24:23.233612 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:24:23.233615 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:24:23.233616 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:24:23.233687 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:24:23.233978 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:24:23.233981 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:24:23.233983 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:24:23.234173 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:24:23.234333 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0808 22:24:23.234929 (+ 596us) server_negotiation.cc:299] Negotiation successful
0808 22:24:23.235022 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260808 22:24:23.237267 30689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86db710979c54485a00d238bccfc957e: Bootstrap starting.
I20260808 22:24:23.237860 30578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e88c02461e15448f966d535d8ea2f340: Bootstrap starting.
I20260808 22:24:23.238785 9923 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 22:24:23.239908 30689 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86db710979c54485a00d238bccfc957e: Neither blocks nor log segments found. Creating new log.
I20260808 22:24:23.240126 30470 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd71bc7679e442548dedaff93eb3363b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd71bc7679e442548dedaff93eb3363b" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 33445 } } peers { permanent_uuid: "e88c02461e15448f966d535d8ea2f340" member_type: VOTER last_known_addr { host: "127.9.176.253" port: 34561 } } peers { permanent_uuid: "86db710979c54485a00d238bccfc957e" member_type: VOTER last_known_addr { host: "127.9.176.252" port: 41631 } }
I20260808 22:24:23.240332 30578 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e88c02461e15448f966d535d8ea2f340: Neither blocks nor log segments found. Creating new log.
I20260808 22:24:23.240370 30470 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd71bc7679e442548dedaff93eb3363b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:24:23.240417 30470 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd71bc7679e442548dedaff93eb3363b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd71bc7679e442548dedaff93eb3363b, State: Initialized, Role: FOLLOWER
I20260808 22:24:23.240809 30689 log.cc:824] T 00000000000000000000000000000000 P 86db710979c54485a00d238bccfc957e: Log is configured to *not* fsync() on all Append() calls
I20260808 22:24:23.240860 30470 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd71bc7679e442548dedaff93eb3363b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd71bc7679e442548dedaff93eb3363b" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 33445 } } peers { permanent_uuid: "e88c02461e15448f966d535d8ea2f340" member_type: VOTER last_known_addr { host: "127.9.176.253" port: 34561 } } peers { permanent_uuid: "86db710979c54485a00d238bccfc957e" member_type: VOTER last_known_addr { host: "127.9.176.252" port: 41631 } }
I20260808 22:24:23.241250 30578 log.cc:824] T 00000000000000000000000000000000 P e88c02461e15448f966d535d8ea2f340: Log is configured to *not* fsync() on all Append() calls
I20260808 22:24:23.241647 30709 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd71bc7679e442548dedaff93eb3363b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd71bc7679e442548dedaff93eb3363b" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 33445 } } peers { permanent_uuid: "e88c02461e15448f966d535d8ea2f340" member_type: VOTER last_known_addr { host: "127.9.176.253" port: 34561 } } peers { permanent_uuid: "86db710979c54485a00d238bccfc957e" member_type: VOTER last_known_addr { host: "127.9.176.252" port: 41631 } } }
I20260808 22:24:23.241850 30709 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd71bc7679e442548dedaff93eb3363b [sys.catalog]: This master's current role is: FOLLOWER
I20260808 22:24:23.242527 30470 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd71bc7679e442548dedaff93eb3363b [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:24:23.246495 30689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86db710979c54485a00d238bccfc957e: No bootstrap required, opened a new log
W20260808 22:24:23.248420 30406 tablet.cc:2369] T 00000000000000000000000000000000 P fd71bc7679e442548dedaff93eb3363b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 22:24:23.249009 30578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e88c02461e15448f966d535d8ea2f340: No bootstrap required, opened a new log
I20260808 22:24:23.251102 30689 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86db710979c54485a00d238bccfc957e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd71bc7679e442548dedaff93eb3363b" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 33445 } } peers { permanent_uuid: "e88c02461e15448f966d535d8ea2f340" member_type: VOTER last_known_addr { host: "127.9.176.253" port: 34561 } } peers { permanent_uuid: "86db710979c54485a00d238bccfc957e" member_type: VOTER last_known_addr { host: "127.9.176.252" port: 41631 } }
I20260808 22:24:23.251305 30693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:23.241194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38230 (local address 127.9.176.252:41631)
0808 22:24:23.241733 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0808 22:24:23.241736 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:24:23.242106 (+ 370us) server_negotiation.cc:407] Connection header received
0808 22:24:23.242236 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:24:23.242239 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:24:23.242283 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:24:23.242367 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:24:23.243183 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:23.243929 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:23.245045 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:23.245371 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:23.245438 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:23.247150 (+ 1712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:24:23.247154 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:24:23.247157 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:24:23.247421 (+ 264us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:24:23.249465 (+ 2044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:24:23.249951 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:24:23.249955 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:24:23.249956 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:24:23.250042 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:24:23.250352 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:24:23.250355 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:24:23.250357 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:24:23.250594 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:24:23.250696 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0808 22:24:23.251046 (+ 350us) server_negotiation.cc:299] Negotiation successful
0808 22:24:23.251158 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488}
I20260808 22:24:23.251336 30689 raft_consensus.cc:374] T 00000000000000000000000000000000 P 86db710979c54485a00d238bccfc957e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:24:23.251396 30689 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86db710979c54485a00d238bccfc957e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86db710979c54485a00d238bccfc957e, State: Initialized, Role: FOLLOWER
I20260808 22:24:23.251662 30589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:23.242207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58110 (local address 127.9.176.253:34561)
0808 22:24:23.242307 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0808 22:24:23.242309 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:24:23.242791 (+ 482us) server_negotiation.cc:407] Connection header received
0808 22:24:23.242960 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:24:23.242963 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:24:23.243007 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:24:23.243069 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:24:23.243911 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:23.244658 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:23.246360 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:23.246692 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:23.246771 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:23.248315 (+ 1544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:24:23.248319 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:24:23.248321 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:24:23.248346 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:24:23.249870 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:24:23.250350 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:24:23.250354 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:24:23.250356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:24:23.250441 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:24:23.250747 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:24:23.250751 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:24:23.250752 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:24:23.250952 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:24:23.251043 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0808 22:24:23.251419 (+ 376us) server_negotiation.cc:299] Negotiation successful
0808 22:24:23.251531 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260808 22:24:23.252420 30689 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86db710979c54485a00d238bccfc957e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd71bc7679e442548dedaff93eb3363b" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 33445 } } peers { permanent_uuid: "e88c02461e15448f966d535d8ea2f340" member_type: VOTER last_known_addr { host: "127.9.176.253" port: 34561 } } peers { permanent_uuid: "86db710979c54485a00d238bccfc957e" member_type: VOTER last_known_addr { host: "127.9.176.252" port: 41631 } }
I20260808 22:24:23.252986 30578 raft_consensus.cc:348] T 00000000000000000000000000000000 P e88c02461e15448f966d535d8ea2f340 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd71bc7679e442548dedaff93eb3363b" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 33445 } } peers { permanent_uuid: "e88c02461e15448f966d535d8ea2f340" member_type: VOTER last_known_addr { host: "127.9.176.253" port: 34561 } } peers { permanent_uuid: "86db710979c54485a00d238bccfc957e" member_type: VOTER last_known_addr { host: "127.9.176.252" port: 41631 } }
I20260808 22:24:23.253261 30578 raft_consensus.cc:374] T 00000000000000000000000000000000 P e88c02461e15448f966d535d8ea2f340 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:24:23.253360 30578 raft_consensus.cc:729] T 00000000000000000000000000000000 P e88c02461e15448f966d535d8ea2f340 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e88c02461e15448f966d535d8ea2f340, State: Initialized, Role: FOLLOWER
I20260808 22:24:23.253839 30578 consensus_queue.cc:260] T 00000000000000000000000000000000 P e88c02461e15448f966d535d8ea2f340 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd71bc7679e442548dedaff93eb3363b" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 33445 } } peers { permanent_uuid: "e88c02461e15448f966d535d8ea2f340" member_type: VOTER last_known_addr { host: "127.9.176.253" port: 34561 } } peers { permanent_uuid: "86db710979c54485a00d238bccfc957e" member_type: VOTER last_known_addr { host: "127.9.176.252" port: 41631 } }
I20260808 22:24:23.255172 30735 sys_catalog.cc:455] T 00000000000000000000000000000000 P e88c02461e15448f966d535d8ea2f340 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd71bc7679e442548dedaff93eb3363b" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 33445 } } peers { permanent_uuid: "e88c02461e15448f966d535d8ea2f340" member_type: VOTER last_known_addr { host: "127.9.176.253" port: 34561 } } peers { permanent_uuid: "86db710979c54485a00d238bccfc957e" member_type: VOTER last_known_addr { host: "127.9.176.252" port: 41631 } } }
I20260808 22:24:23.255287 30578 sys_catalog.cc:564] T 00000000000000000000000000000000 P e88c02461e15448f966d535d8ea2f340 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:24:23.255359 30735 sys_catalog.cc:458] T 00000000000000000000000000000000 P e88c02461e15448f966d535d8ea2f340 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 22:24:23.252790 30377 master_runner.cc:186] Error getting master registration for 127.9.176.253:34561: OK, instance_id { permanent_uuid: "e88c02461e15448f966d535d8ea2f340" instance_seqno: 1786227862973025 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 22:24:23.256731 30377 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260808 22:24:23.257387 30734 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fd71bc7679e442548dedaff93eb3363b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 22:24:23.257493 30734 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 22:24:23.258370 30689 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86db710979c54485a00d238bccfc957e [sys.catalog]: configured and running, proceeding with master startup.
W20260808 22:24:23.260088 30753 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e88c02461e15448f966d535d8ea2f340: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 22:24:23.260218 30753 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 22:24:23.264453 30473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:23.241565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46444 (local address 127.9.176.254:33445)
0808 22:24:23.241675 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0808 22:24:23.241679 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:24:23.242758 (+ 1079us) server_negotiation.cc:407] Connection header received
0808 22:24:23.242926 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:24:23.242929 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:24:23.242970 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:24:23.243017 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:24:23.243937 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:23.244628 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:23.256831 (+ 12203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:23.257174 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:23.257276 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:23.260093 (+ 2817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:24:23.260098 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:24:23.260100 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:24:23.260128 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:24:23.261767 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:24:23.262295 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:24:23.262299 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:24:23.262301 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:24:23.262872 (+ 571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:24:23.263193 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:24:23.263196 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:24:23.263198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:24:23.263418 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:24:23.263539 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0808 22:24:23.264177 (+ 638us) server_negotiation.cc:299] Negotiation successful
0808 22:24:23.264298 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260808 22:24:23.265285 30775 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 86db710979c54485a00d238bccfc957e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 22:24:23.265348 30775 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 22:24:23.265606 30756 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86db710979c54485a00d238bccfc957e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd71bc7679e442548dedaff93eb3363b" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 33445 } } peers { permanent_uuid: "e88c02461e15448f966d535d8ea2f340" member_type: VOTER last_known_addr { host: "127.9.176.253" port: 34561 } } peers { permanent_uuid: "86db710979c54485a00d238bccfc957e" member_type: VOTER last_known_addr { host: "127.9.176.252" port: 41631 } } }
I20260808 22:24:23.265821 30756 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86db710979c54485a00d238bccfc957e [sys.catalog]: This master's current role is: FOLLOWER
I20260808 22:24:23.262296 30479 master_runner.cc:186] Error getting master registration for 127.9.176.252:41631: OK, instance_id { permanent_uuid: "86db710979c54485a00d238bccfc957e" instance_seqno: 1786227863190085 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 22:24:23.267860 30479 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 22:24:23.270367 30593 master_runner.cc:263] No leader master found from master 86db710979c54485a00d238bccfc957e
I20260808 22:24:23.270428 30593 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 22:24:23.384930 30735 raft_consensus.cc:482] T 00000000000000000000000000000000 P e88c02461e15448f966d535d8ea2f340 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 22:24:23.385181 30735 raft_consensus.cc:504] T 00000000000000000000000000000000 P e88c02461e15448f966d535d8ea2f340 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd71bc7679e442548dedaff93eb3363b" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 33445 } } peers { permanent_uuid: "e88c02461e15448f966d535d8ea2f340" member_type: VOTER last_known_addr { host: "127.9.176.253" port: 34561 } } peers { permanent_uuid: "86db710979c54485a00d238bccfc957e" member_type: VOTER last_known_addr { host: "127.9.176.252" port: 41631 } }
I20260808 22:24:23.386339 30735 leader_election.cc:288] T 00000000000000000000000000000000 P e88c02461e15448f966d535d8ea2f340 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd71bc7679e442548dedaff93eb3363b (127.9.176.254:33445), 86db710979c54485a00d238bccfc957e (127.9.176.252:41631)
I20260808 22:24:23.387606 30652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e88c02461e15448f966d535d8ea2f340" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86db710979c54485a00d238bccfc957e" is_pre_election: true
I20260808 22:24:23.387965 30652 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 86db710979c54485a00d238bccfc957e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e88c02461e15448f966d535d8ea2f340 in term 0.
I20260808 22:24:23.388576 30505 leader_election.cc:302] T 00000000000000000000000000000000 P e88c02461e15448f966d535d8ea2f340 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86db710979c54485a00d238bccfc957e, e88c02461e15448f966d535d8ea2f340; no voters:
I20260808 22:24:23.389127 30434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e88c02461e15448f966d535d8ea2f340" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd71bc7679e442548dedaff93eb3363b" is_pre_election: true
I20260808 22:24:23.389401 30434 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fd71bc7679e442548dedaff93eb3363b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e88c02461e15448f966d535d8ea2f340 in term 0.
I20260808 22:24:23.390028 30735 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e88c02461e15448f966d535d8ea2f340 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 22:24:23.390108 30735 raft_consensus.cc:482] T 00000000000000000000000000000000 P e88c02461e15448f966d535d8ea2f340 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 22:24:23.390144 30735 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e88c02461e15448f966d535d8ea2f340 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:24:23.391367 30735 raft_consensus.cc:504] T 00000000000000000000000000000000 P e88c02461e15448f966d535d8ea2f340 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd71bc7679e442548dedaff93eb3363b" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 33445 } } peers { permanent_uuid: "e88c02461e15448f966d535d8ea2f340" member_type: VOTER last_known_addr { host: "127.9.176.253" port: 34561 } } peers { permanent_uuid: "86db710979c54485a00d238bccfc957e" member_type: VOTER last_known_addr { host: "127.9.176.252" port: 41631 } }
I20260808 22:24:23.391695 30735 leader_election.cc:288] T 00000000000000000000000000000000 P e88c02461e15448f966d535d8ea2f340 [CANDIDATE]: Term 1 election: Requested vote from peers fd71bc7679e442548dedaff93eb3363b (127.9.176.254:33445), 86db710979c54485a00d238bccfc957e (127.9.176.252:41631)
I20260808 22:24:23.392344 30652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e88c02461e15448f966d535d8ea2f340" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86db710979c54485a00d238bccfc957e"
I20260808 22:24:23.392457 30652 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86db710979c54485a00d238bccfc957e [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:24:23.393806 30652 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 86db710979c54485a00d238bccfc957e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e88c02461e15448f966d535d8ea2f340 in term 1.
I20260808 22:24:23.394152 30505 leader_election.cc:302] T 00000000000000000000000000000000 P e88c02461e15448f966d535d8ea2f340 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86db710979c54485a00d238bccfc957e, e88c02461e15448f966d535d8ea2f340; no voters:
I20260808 22:24:23.394424 30434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e88c02461e15448f966d535d8ea2f340" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd71bc7679e442548dedaff93eb3363b"
I20260808 22:24:23.394552 30434 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd71bc7679e442548dedaff93eb3363b [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:24:23.395761 30434 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fd71bc7679e442548dedaff93eb3363b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e88c02461e15448f966d535d8ea2f340 in term 1.
I20260808 22:24:23.396231 30735 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e88c02461e15448f966d535d8ea2f340 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:24:23.396497 30735 raft_consensus.cc:686] T 00000000000000000000000000000000 P e88c02461e15448f966d535d8ea2f340 [term 1 LEADER]: Becoming Leader. State: Replica: e88c02461e15448f966d535d8ea2f340, State: Running, Role: LEADER
I20260808 22:24:23.396908 30735 consensus_queue.cc:237] T 00000000000000000000000000000000 P e88c02461e15448f966d535d8ea2f340 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd71bc7679e442548dedaff93eb3363b" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 33445 } } peers { permanent_uuid: "e88c02461e15448f966d535d8ea2f340" member_type: VOTER last_known_addr { host: "127.9.176.253" port: 34561 } } peers { permanent_uuid: "86db710979c54485a00d238bccfc957e" member_type: VOTER last_known_addr { host: "127.9.176.252" port: 41631 } }
I20260808 22:24:23.398759 30735 sys_catalog.cc:455] T 00000000000000000000000000000000 P e88c02461e15448f966d535d8ea2f340 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e88c02461e15448f966d535d8ea2f340. Latest consensus state: current_term: 1 leader_uuid: "e88c02461e15448f966d535d8ea2f340" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd71bc7679e442548dedaff93eb3363b" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 33445 } } peers { permanent_uuid: "e88c02461e15448f966d535d8ea2f340" member_type: VOTER last_known_addr { host: "127.9.176.253" port: 34561 } } peers { permanent_uuid: "86db710979c54485a00d238bccfc957e" member_type: VOTER last_known_addr { host: "127.9.176.252" port: 41631 } } }
I20260808 22:24:23.401520 30735 sys_catalog.cc:458] T 00000000000000000000000000000000 P e88c02461e15448f966d535d8ea2f340 [sys.catalog]: This master's current role is: LEADER
I20260808 22:24:23.403445 30781 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:24:23.404369 30781 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:24:23.409988 30652 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 86db710979c54485a00d238bccfc957e [term 1 FOLLOWER]: Refusing update from remote peer e88c02461e15448f966d535d8ea2f340: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:24:23.410915 30778 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e88c02461e15448f966d535d8ea2f340 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86db710979c54485a00d238bccfc957e" member_type: VOTER last_known_addr { host: "127.9.176.252" port: 41631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:24:23.415580 30756 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86db710979c54485a00d238bccfc957e [sys.catalog]: SysCatalogTable state changed. Reason: New leader e88c02461e15448f966d535d8ea2f340. Latest consensus state: current_term: 1 leader_uuid: "e88c02461e15448f966d535d8ea2f340" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd71bc7679e442548dedaff93eb3363b" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 33445 } } peers { permanent_uuid: "e88c02461e15448f966d535d8ea2f340" member_type: VOTER last_known_addr { host: "127.9.176.253" port: 34561 } } peers { permanent_uuid: "86db710979c54485a00d238bccfc957e" member_type: VOTER last_known_addr { host: "127.9.176.252" port: 41631 } } }
I20260808 22:24:23.415746 30756 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86db710979c54485a00d238bccfc957e [sys.catalog]: This master's current role is: FOLLOWER
I20260808 22:24:23.419564 30434 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fd71bc7679e442548dedaff93eb3363b [term 1 FOLLOWER]: Refusing update from remote peer e88c02461e15448f966d535d8ea2f340: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:24:23.420234 30778 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e88c02461e15448f966d535d8ea2f340 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd71bc7679e442548dedaff93eb3363b" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 33445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:24:23.424230 30709 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd71bc7679e442548dedaff93eb3363b [sys.catalog]: SysCatalogTable state changed. Reason: New leader e88c02461e15448f966d535d8ea2f340. Latest consensus state: current_term: 1 leader_uuid: "e88c02461e15448f966d535d8ea2f340" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd71bc7679e442548dedaff93eb3363b" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 33445 } } peers { permanent_uuid: "e88c02461e15448f966d535d8ea2f340" member_type: VOTER last_known_addr { host: "127.9.176.253" port: 34561 } } peers { permanent_uuid: "86db710979c54485a00d238bccfc957e" member_type: VOTER last_known_addr { host: "127.9.176.252" port: 41631 } } }
I20260808 22:24:23.424364 30709 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd71bc7679e442548dedaff93eb3363b [sys.catalog]: This master's current role is: FOLLOWER
I20260808 22:24:23.427158 30778 sys_catalog.cc:455] T 00000000000000000000000000000000 P e88c02461e15448f966d535d8ea2f340 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e88c02461e15448f966d535d8ea2f340" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd71bc7679e442548dedaff93eb3363b" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 33445 } } peers { permanent_uuid: "e88c02461e15448f966d535d8ea2f340" member_type: VOTER last_known_addr { host: "127.9.176.253" port: 34561 } } peers { permanent_uuid: "86db710979c54485a00d238bccfc957e" member_type: VOTER last_known_addr { host: "127.9.176.252" port: 41631 } } }
I20260808 22:24:23.427285 30778 sys_catalog.cc:458] T 00000000000000000000000000000000 P e88c02461e15448f966d535d8ea2f340 [sys.catalog]: This master's current role is: LEADER
I20260808 22:24:23.428357 30735 sys_catalog.cc:455] T 00000000000000000000000000000000 P e88c02461e15448f966d535d8ea2f340 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e88c02461e15448f966d535d8ea2f340" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd71bc7679e442548dedaff93eb3363b" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 33445 } } peers { permanent_uuid: "e88c02461e15448f966d535d8ea2f340" member_type: VOTER last_known_addr { host: "127.9.176.253" port: 34561 } } peers { permanent_uuid: "86db710979c54485a00d238bccfc957e" member_type: VOTER last_known_addr { host: "127.9.176.252" port: 41631 } } }
I20260808 22:24:23.428470 30735 sys_catalog.cc:458] T 00000000000000000000000000000000 P e88c02461e15448f966d535d8ea2f340 [sys.catalog]: This master's current role is: LEADER
I20260808 22:24:23.428648 30756 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86db710979c54485a00d238bccfc957e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e88c02461e15448f966d535d8ea2f340" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd71bc7679e442548dedaff93eb3363b" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 33445 } } peers { permanent_uuid: "e88c02461e15448f966d535d8ea2f340" member_type: VOTER last_known_addr { host: "127.9.176.253" port: 34561 } } peers { permanent_uuid: "86db710979c54485a00d238bccfc957e" member_type: VOTER last_known_addr { host: "127.9.176.252" port: 41631 } } }
I20260808 22:24:23.428761 30756 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86db710979c54485a00d238bccfc957e [sys.catalog]: This master's current role is: FOLLOWER
I20260808 22:24:23.429451 30709 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd71bc7679e442548dedaff93eb3363b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e88c02461e15448f966d535d8ea2f340" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd71bc7679e442548dedaff93eb3363b" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 33445 } } peers { permanent_uuid: "e88c02461e15448f966d535d8ea2f340" member_type: VOTER last_known_addr { host: "127.9.176.253" port: 34561 } } peers { permanent_uuid: "86db710979c54485a00d238bccfc957e" member_type: VOTER last_known_addr { host: "127.9.176.252" port: 41631 } } }
I20260808 22:24:23.429593 30709 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd71bc7679e442548dedaff93eb3363b [sys.catalog]: This master's current role is: FOLLOWER
I20260808 22:24:23.429708 30782 mvcc.cc:205] Tried to move back new op lower bound from 7316389328519651328 to 7316389328477155328. Current Snapshot: MvccSnapshot[applied={T|T < 7316389328519651328}]
I20260808 22:24:23.429840 30784 mvcc.cc:205] Tried to move back new op lower bound from 7316389328519651328 to 7316389328477155328. Current Snapshot: MvccSnapshot[applied={T|T < 7316389328519651328}]
I20260808 22:24:23.429838 30787 mvcc.cc:205] Tried to move back new op lower bound from 7316389328519651328 to 7316389328477155328. Current Snapshot: MvccSnapshot[applied={T|T < 7316389328519651328}]
I20260808 22:24:23.431520 30781 catalog_manager.cc:1347] Generated new cluster ID: ee5adc8e889846baa4f6767c132a4f33
I20260808 22:24:23.431587 30781 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:24:23.454155 30781 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:24:23.455318 30781 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:24:23.474802 30781 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e88c02461e15448f966d535d8ea2f340: Generated new TSK 0
I20260808 22:24:23.475492 30781 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 22:24:24.260269 30734 catalog_manager.cc:1259] Loaded cluster ID: ee5adc8e889846baa4f6767c132a4f33
I20260808 22:24:24.260357 30734 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fd71bc7679e442548dedaff93eb3363b: loading cluster ID for follower catalog manager: success
I20260808 22:24:24.262533 30734 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fd71bc7679e442548dedaff93eb3363b: acquiring CA information for follower catalog manager: success
I20260808 22:24:24.263908 30734 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fd71bc7679e442548dedaff93eb3363b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 22:24:24.266428 30775 catalog_manager.cc:1259] Loaded cluster ID: ee5adc8e889846baa4f6767c132a4f33
I20260808 22:24:24.266494 30775 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 86db710979c54485a00d238bccfc957e: loading cluster ID for follower catalog manager: success
I20260808 22:24:24.268507 30775 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 86db710979c54485a00d238bccfc957e: acquiring CA information for follower catalog manager: success
I20260808 22:24:24.269764 30775 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 86db710979c54485a00d238bccfc957e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 22:24:24.529590 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 30377
I20260808 22:24:24.542114 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 30479
I20260808 22:24:24.551441 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 30593
2026-08-08T22:24:24Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3192 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786227706282457-9923-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 22:24:24 dist-test-slave-ncp9 krb5kdc[30800](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 22:24:24 dist-test-slave-ncp9 krb5kdc[30800](info): set up 2 sockets
Aug 08 22:24:24 dist-test-slave-ncp9 krb5kdc[30800](info): commencing operation
krb5kdc: starting...
W20260808 22:24:25.676991 9923 mini_kdc.cc:121] Time spent starting KDC: real 1.102s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786227706282457-9923-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786227706282457-9923-0/krb5kdc/test-user.keytab.
Aug 08 22:24:25 dist-test-slave-ncp9 krb5kdc[30800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227865, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T22:24:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:24:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.176.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.176.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.176.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab.
Entry for principal kudu/127.9.176.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab.
I20260808 22:24:25.793634 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786227706282457-9923-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786227706282457-9923-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786227706282457-9923-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:38697
--webserver_interface=127.9.176.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab
--principal=kudu/127.9.176.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:43079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.176.254:38697
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:24:25.901199 30821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:24:25.901439 30821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:24:25.901479 30821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:24:25.905371 30821 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:24:25.905421 30821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:24:25.905438 30821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:24:25.905469 30821 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:24:25.905494 30821 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:24:25.910502 30821 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:43079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786227706282457-9923-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786227706282457-9923-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.176.254:38697
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab
--principal=kudu/127.9.176.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:38697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.176.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30821
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786227706282457-9923-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:24:25.911448 30821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:24:25.912395 30821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:24:25.920452 30831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:24:25.920499 30829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:24:25.920456 30833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:24:25.920938 30821 server_base.cc:1034] running on GCE node
I20260808 22:24:25.921461 30821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:24:25.922537 30821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:24:25.923699 30821 hybrid_clock.cc:648] HybridClock initialized: now 1786227865923685 us; error 28 us; skew 500 ppm
Aug 08 22:24:25 dist-test-slave-ncp9 krb5kdc[30800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227865, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:24:25.928633 30821 init.cc:377] Logged in from keytab as kudu/127.9.176.254@KRBTEST.COM (short username kudu)
I20260808 22:24:25.929512 30821 webserver.cc:460] Webserver started at http://127.9.176.254:45633/ using document root <none> and password file <none>
I20260808 22:24:25.930038 30821 fs_manager.cc:359] Metadata directory not provided
I20260808 22:24:25.930104 30821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:24:25.930282 30821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:24:25.932260 30821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786227706282457-9923-0/minicluster-data/master-0/data/instance:
uuid: "0c34436c6b17428694ff812a46a02d00"
format_stamp: "Formatted at 2026-08-08 22:24:25 on dist-test-slave-ncp9"
server_key: "7d541fadf1d398da277a77c53274832c"
server_key_iv: "da0712bc1ddfd4a5edcd00efb116bb0d"
server_key_version: "encryptionkey@0"
I20260808 22:24:25.932796 30821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786227706282457-9923-0/minicluster-data/master-0/wal/instance:
uuid: "0c34436c6b17428694ff812a46a02d00"
format_stamp: "Formatted at 2026-08-08 22:24:25 on dist-test-slave-ncp9"
server_key: "7d541fadf1d398da277a77c53274832c"
server_key_iv: "da0712bc1ddfd4a5edcd00efb116bb0d"
server_key_version: "encryptionkey@0"
I20260808 22:24:25.936231 30821 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260808 22:24:25.938504 30842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:24:25.939615 30821 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 22:24:25.939739 30821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786227706282457-9923-0/minicluster-data/master-0/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786227706282457-9923-0/minicluster-data/master-0/wal
uuid: "0c34436c6b17428694ff812a46a02d00"
format_stamp: "Formatted at 2026-08-08 22:24:25 on dist-test-slave-ncp9"
server_key: "7d541fadf1d398da277a77c53274832c"
server_key_iv: "da0712bc1ddfd4a5edcd00efb116bb0d"
server_key_version: "encryptionkey@0"
I20260808 22:24:25.939831 30821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786227706282457-9923-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786227706282457-9923-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786227706282457-9923-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:24:25.953210 30821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:24:25.955716 30821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:24:25.955883 30821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:24:25.962793 30821 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.254:38697
I20260808 22:24:25.962806 30912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.254:38697 every 8 connection(s)
I20260808 22:24:25.963730 30821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
I20260808 22:24:25.966436 30915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:24:25.968837 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 30821
I20260808 22:24:25.968986 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786227706282457-9923-0/minicluster-data/master-0/wal/instance
I20260808 22:24:25.969275 9923 external_mini_cluster.cc:1428] Setting key 577e3587dbf9b2f00d505def185ea906
I20260808 22:24:25.972277 30915 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c34436c6b17428694ff812a46a02d00: Bootstrap starting.
I20260808 22:24:25.974730 30915 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c34436c6b17428694ff812a46a02d00: Neither blocks nor log segments found. Creating new log.
I20260808 22:24:25.975549 30915 log.cc:824] T 00000000000000000000000000000000 P 0c34436c6b17428694ff812a46a02d00: Log is configured to *not* fsync() on all Append() calls
I20260808 22:24:25.977494 30915 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c34436c6b17428694ff812a46a02d00: No bootstrap required, opened a new log
Aug 08 22:24:25 dist-test-slave-ncp9 krb5kdc[30800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227865, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
I20260808 22:24:25.980552 30915 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c34436c6b17428694ff812a46a02d00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c34436c6b17428694ff812a46a02d00" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 38697 } }
I20260808 22:24:25.980760 30915 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c34436c6b17428694ff812a46a02d00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:24:25.980805 30915 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c34436c6b17428694ff812a46a02d00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c34436c6b17428694ff812a46a02d00, State: Initialized, Role: FOLLOWER
I20260808 22:24:25.981247 30915 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c34436c6b17428694ff812a46a02d00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c34436c6b17428694ff812a46a02d00" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 38697 } }
I20260808 22:24:25.981370 30915 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c34436c6b17428694ff812a46a02d00 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:24:25.981431 30915 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c34436c6b17428694ff812a46a02d00 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:24:25.981520 30915 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c34436c6b17428694ff812a46a02d00 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:24:25.982322 30915 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c34436c6b17428694ff812a46a02d00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c34436c6b17428694ff812a46a02d00" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 38697 } }
I20260808 22:24:25.982664 30915 leader_election.cc:302] T 00000000000000000000000000000000 P 0c34436c6b17428694ff812a46a02d00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c34436c6b17428694ff812a46a02d00; no voters:
I20260808 22:24:25.982939 30915 leader_election.cc:288] T 00000000000000000000000000000000 P 0c34436c6b17428694ff812a46a02d00 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:24:25.983042 30923 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c34436c6b17428694ff812a46a02d00 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:24:25.983211 30923 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c34436c6b17428694ff812a46a02d00 [term 1 LEADER]: Becoming Leader. State: Replica: 0c34436c6b17428694ff812a46a02d00, State: Running, Role: LEADER
I20260808 22:24:25.983438 30920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:25.970562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58822 (local address 127.9.176.254:38697)
0808 22:24:25.970950 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0808 22:24:25.970965 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 22:24:25.970999 (+ 34us) server_negotiation.cc:407] Connection header received
0808 22:24:25.971630 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:24:25.971647 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:24:25.971966 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:24:25.972244 (+ 278us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:24:25.973087 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:25.974070 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:25.975192 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:25.975714 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:25.975838 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:25.978345 (+ 2507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:24:25.978351 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:24:25.978363 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:24:25.978389 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:24:25.980446 (+ 2057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:24:25.980881 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:24:25.980887 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:24:25.980912 (+ 25us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:24:25.981031 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:24:25.981299 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:24:25.981302 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:24:25.981304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:24:25.981766 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:24:25.981953 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0808 22:24:25.982241 (+ 288us) server_negotiation.cc:299] Negotiation successful
0808 22:24:25.982538 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":109,"threads_started":1}
I20260808 22:24:25.983567 30923 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c34436c6b17428694ff812a46a02d00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c34436c6b17428694ff812a46a02d00" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 38697 } }
I20260808 22:24:25.984156 30915 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c34436c6b17428694ff812a46a02d00 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:24:25.985772 30924 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c34436c6b17428694ff812a46a02d00 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c34436c6b17428694ff812a46a02d00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c34436c6b17428694ff812a46a02d00" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 38697 } } }
I20260808 22:24:25.985893 30924 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c34436c6b17428694ff812a46a02d00 [sys.catalog]: This master's current role is: LEADER
I20260808 22:24:25.986629 9923 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 22:24:25.986538 30925 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c34436c6b17428694ff812a46a02d00 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c34436c6b17428694ff812a46a02d00. Latest consensus state: current_term: 1 leader_uuid: "0c34436c6b17428694ff812a46a02d00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c34436c6b17428694ff812a46a02d00" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 38697 } } }
I20260808 22:24:25.986930 30925 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c34436c6b17428694ff812a46a02d00 [sys.catalog]: This master's current role is: LEADER
W20260808 22:24:25.989475 30945 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0c34436c6b17428694ff812a46a02d00: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 22:24:25.989566 30945 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 22:24:25.989636 30937 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:24:25.990327 30937 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:24:25.995329 30937 catalog_manager.cc:1347] Generated new cluster ID: 018219a496054789ae4a9172c371a0d3
I20260808 22:24:25.995379 30937 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:24:26.011855 30937 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:24:26.012796 30937 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:24:26.020999 30937 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c34436c6b17428694ff812a46a02d00: Generated new TSK 0
I20260808 22:24:26.021631 30937 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 22:24:26.042268 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 30821
2026-08-08T22:24:26Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1494 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 22:24:26 dist-test-slave-ncp9 krb5kdc[30953](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 22:24:26 dist-test-slave-ncp9 krb5kdc[30953](info): set up 2 sockets
Aug 08 22:24:26 dist-test-slave-ncp9 krb5kdc[30953](info): commencing operation
krb5kdc: starting...
W20260808 22:24:27.175118 9923 mini_kdc.cc:121] Time spent starting KDC: real 1.106s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/krb5kdc/test-user.keytab.
Aug 08 22:24:27 dist-test-slave-ncp9 krb5kdc[30953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227867, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T22:24:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:24:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.176.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.176.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.176.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab.
Entry for principal kudu/127.9.176.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab.
I20260808 22:24:27.288575 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:38597
--webserver_interface=127.9.176.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab
--principal=kudu/127.9.176.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:33087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.176.254:38597
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:24:27.395128 30971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:24:27.395377 30971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:24:27.395416 30971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:24:27.399326 30971 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:24:27.399376 30971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:24:27.399394 30971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:24:27.399408 30971 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:24:27.399421 30971 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:24:27.404493 30971 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:33087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.176.254:38597
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab
--principal=kudu/127.9.176.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:38597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.176.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30971
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:24:27.405464 30971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:24:27.406425 30971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:24:27.414376 30978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:24:27.414409 30979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:24:27.414376 30982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:24:27.414678 30971 server_base.cc:1034] running on GCE node
I20260808 22:24:27.415230 30971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:24:27.416234 30971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:24:27.417387 30971 hybrid_clock.cc:648] HybridClock initialized: now 1786227867417364 us; error 37 us; skew 500 ppm
Aug 08 22:24:27 dist-test-slave-ncp9 krb5kdc[30953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227867, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:24:27.419984 30971 init.cc:377] Logged in from keytab as kudu/127.9.176.254@KRBTEST.COM (short username kudu)
I20260808 22:24:27.420800 30971 webserver.cc:460] Webserver started at http://127.9.176.254:40381/ using document root <none> and password file <none>
I20260808 22:24:27.421361 30971 fs_manager.cc:359] Metadata directory not provided
I20260808 22:24:27.421468 30971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:24:27.421684 30971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:24:27.423666 30971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/master-0/data/instance:
uuid: "35e08196664442e9b7f686512baa70f5"
format_stamp: "Formatted at 2026-08-08 22:24:27 on dist-test-slave-ncp9"
server_key: "7653e1e631037b8f7bd49e7fad710b74"
server_key_iv: "86cf734e2347e21a839717c26f3d69a7"
server_key_version: "encryptionkey@0"
I20260808 22:24:27.424220 30971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/master-0/wal/instance:
uuid: "35e08196664442e9b7f686512baa70f5"
format_stamp: "Formatted at 2026-08-08 22:24:27 on dist-test-slave-ncp9"
server_key: "7653e1e631037b8f7bd49e7fad710b74"
server_key_iv: "86cf734e2347e21a839717c26f3d69a7"
server_key_version: "encryptionkey@0"
I20260808 22:24:27.427773 30971 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 22:24:27.430109 30994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:24:27.431308 30971 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 22:24:27.431418 30971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/master-0/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/master-0/wal
uuid: "35e08196664442e9b7f686512baa70f5"
format_stamp: "Formatted at 2026-08-08 22:24:27 on dist-test-slave-ncp9"
server_key: "7653e1e631037b8f7bd49e7fad710b74"
server_key_iv: "86cf734e2347e21a839717c26f3d69a7"
server_key_version: "encryptionkey@0"
I20260808 22:24:27.431514 30971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:24:27.448843 30971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:24:27.452731 30971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:24:27.452957 30971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:24:27.459930 30971 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.254:38597
I20260808 22:24:27.460197 31069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.254:38597 every 8 connection(s)
I20260808 22:24:27.460942 30971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
I20260808 22:24:27.463745 31070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:24:27.463840 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 30971
I20260808 22:24:27.463961 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/master-0/wal/instance
I20260808 22:24:27.464282 9923 external_mini_cluster.cc:1428] Setting key 5c79cbcc1b2951a551feb455875b215e
I20260808 22:24:27.470918 31070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35e08196664442e9b7f686512baa70f5: Bootstrap starting.
I20260808 22:24:27.473349 31070 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 35e08196664442e9b7f686512baa70f5: Neither blocks nor log segments found. Creating new log.
Aug 08 22:24:27 dist-test-slave-ncp9 krb5kdc[30953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227867, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
I20260808 22:24:27.474226 31070 log.cc:824] T 00000000000000000000000000000000 P 35e08196664442e9b7f686512baa70f5: Log is configured to *not* fsync() on all Append() calls
I20260808 22:24:27.476850 31070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35e08196664442e9b7f686512baa70f5: No bootstrap required, opened a new log
I20260808 22:24:27.479451 31076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:27.465616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33432 (local address 127.9.176.254:38597)
0808 22:24:27.466048 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0808 22:24:27.466067 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0808 22:24:27.466110 (+ 43us) server_negotiation.cc:407] Connection header received
0808 22:24:27.467199 (+ 1089us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:24:27.467224 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:24:27.467640 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:24:27.468014 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:24:27.468658 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:27.470173 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:27.470979 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:27.471641 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:27.471795 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:27.474303 (+ 2508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:24:27.474309 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:24:27.474326 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:24:27.474359 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:24:27.476371 (+ 2012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:24:27.476908 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:24:27.476915 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:24:27.476922 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:24:27.477061 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:24:27.477326 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:24:27.477329 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:24:27.477330 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:24:27.477839 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:24:27.478034 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0808 22:24:27.478206 (+ 172us) server_negotiation.cc:299] Negotiation successful
0808 22:24:27.478500 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":101,"threads_started":1}
I20260808 22:24:27.481257 31070 raft_consensus.cc:348] T 00000000000000000000000000000000 P 35e08196664442e9b7f686512baa70f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35e08196664442e9b7f686512baa70f5" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 38597 } }
I20260808 22:24:27.481485 31070 raft_consensus.cc:374] T 00000000000000000000000000000000 P 35e08196664442e9b7f686512baa70f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:24:27.481544 31070 raft_consensus.cc:729] T 00000000000000000000000000000000 P 35e08196664442e9b7f686512baa70f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35e08196664442e9b7f686512baa70f5, State: Initialized, Role: FOLLOWER
I20260808 22:24:27.482003 31070 consensus_queue.cc:260] T 00000000000000000000000000000000 P 35e08196664442e9b7f686512baa70f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35e08196664442e9b7f686512baa70f5" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 38597 } }
I20260808 22:24:27.482127 31070 raft_consensus.cc:388] T 00000000000000000000000000000000 P 35e08196664442e9b7f686512baa70f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:24:27.482172 31070 raft_consensus.cc:482] T 00000000000000000000000000000000 P 35e08196664442e9b7f686512baa70f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:24:27.482257 31070 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 35e08196664442e9b7f686512baa70f5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:24:27.483054 31070 raft_consensus.cc:504] T 00000000000000000000000000000000 P 35e08196664442e9b7f686512baa70f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35e08196664442e9b7f686512baa70f5" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 38597 } }
I20260808 22:24:27.483354 31070 leader_election.cc:302] T 00000000000000000000000000000000 P 35e08196664442e9b7f686512baa70f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 35e08196664442e9b7f686512baa70f5; no voters:
I20260808 22:24:27.483632 31070 leader_election.cc:288] T 00000000000000000000000000000000 P 35e08196664442e9b7f686512baa70f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:24:27.483717 31078 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 35e08196664442e9b7f686512baa70f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:24:27.483922 31078 raft_consensus.cc:686] T 00000000000000000000000000000000 P 35e08196664442e9b7f686512baa70f5 [term 1 LEADER]: Becoming Leader. State: Replica: 35e08196664442e9b7f686512baa70f5, State: Running, Role: LEADER
I20260808 22:24:27.484248 31078 consensus_queue.cc:237] T 00000000000000000000000000000000 P 35e08196664442e9b7f686512baa70f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35e08196664442e9b7f686512baa70f5" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 38597 } }
I20260808 22:24:27.484776 31070 sys_catalog.cc:564] T 00000000000000000000000000000000 P 35e08196664442e9b7f686512baa70f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:24:27.486184 31080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35e08196664442e9b7f686512baa70f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 35e08196664442e9b7f686512baa70f5. Latest consensus state: current_term: 1 leader_uuid: "35e08196664442e9b7f686512baa70f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35e08196664442e9b7f686512baa70f5" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 38597 } } }
I20260808 22:24:27.486310 31080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35e08196664442e9b7f686512baa70f5 [sys.catalog]: This master's current role is: LEADER
I20260808 22:24:27.486941 31079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35e08196664442e9b7f686512baa70f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "35e08196664442e9b7f686512baa70f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35e08196664442e9b7f686512baa70f5" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 38597 } } }
I20260808 22:24:27.487066 31093 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:24:27.487282 31079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35e08196664442e9b7f686512baa70f5 [sys.catalog]: This master's current role is: LEADER
I20260808 22:24:27.489681 31093 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:24:27.494926 31093 catalog_manager.cc:1347] Generated new cluster ID: 9f79495c212b4ff4a668128b2376b2af
I20260808 22:24:27.494980 31093 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:24:27.502506 31093 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:24:27.503450 31093 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:24:27.514319 31093 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 35e08196664442e9b7f686512baa70f5: Generated new TSK 0
I20260808 22:24:27.515002 31093 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.176.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.176.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.176.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.193.keytab.
Entry for principal kudu/127.9.176.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.193.keytab.
I20260808 22:24:27.547156 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.193:0
--local_ip_for_outbound_sockets=127.9.176.193
--webserver_interface=127.9.176.193
--webserver_port=0
--tserver_master_addrs=127.9.176.254:38597
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.193.keytab
--principal=kudu/127.9.176.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:33087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:24:27.658551 31109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:24:27.658811 31109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:24:27.658852 31109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:24:27.662813 31109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:24:27.662863 31109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:24:27.662931 31109 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.193
I20260808 22:24:27.667979 31109 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:33087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.193.keytab
--principal=kudu/127.9.176.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.176.193
--webserver_port=0
--tserver_master_addrs=127.9.176.254:38597
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31109
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.193
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:24:27.669028 31109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:24:27.670049 31109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:24:27.679426 31121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:24:27.679535 31119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:24:27.679410 31126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:24:27.679574 31109 server_base.cc:1034] running on GCE node
I20260808 22:24:27.680109 31109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:24:27.680733 31109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:24:27.681908 31109 hybrid_clock.cc:648] HybridClock initialized: now 1786227867681897 us; error 46 us; skew 500 ppm
Aug 08 22:24:27 dist-test-slave-ncp9 krb5kdc[30953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227867, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:24:27.684789 31109 init.cc:377] Logged in from keytab as kudu/127.9.176.193@KRBTEST.COM (short username kudu)
I20260808 22:24:27.685710 31109 webserver.cc:460] Webserver started at http://127.9.176.193:42243/ using document root <none> and password file <none>
I20260808 22:24:27.686292 31109 fs_manager.cc:359] Metadata directory not provided
I20260808 22:24:27.686364 31109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:24:27.686589 31109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:24:27.688570 31109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-0/data/instance:
uuid: "c725f56e88ba4e649a6e854587c1da74"
format_stamp: "Formatted at 2026-08-08 22:24:27 on dist-test-slave-ncp9"
server_key: "c52a2e17aa4900cef485315f7c3cbfe0"
server_key_iv: "abb258a6a786adb289fd93f8bd4a7462"
server_key_version: "encryptionkey@0"
I20260808 22:24:27.689149 31109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-0/wal/instance:
uuid: "c725f56e88ba4e649a6e854587c1da74"
format_stamp: "Formatted at 2026-08-08 22:24:27 on dist-test-slave-ncp9"
server_key: "c52a2e17aa4900cef485315f7c3cbfe0"
server_key_iv: "abb258a6a786adb289fd93f8bd4a7462"
server_key_version: "encryptionkey@0"
I20260808 22:24:27.692728 31109 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260808 22:24:27.695163 31134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:24:27.696404 31109 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 22:24:27.696533 31109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-0/wal
uuid: "c725f56e88ba4e649a6e854587c1da74"
format_stamp: "Formatted at 2026-08-08 22:24:27 on dist-test-slave-ncp9"
server_key: "c52a2e17aa4900cef485315f7c3cbfe0"
server_key_iv: "abb258a6a786adb289fd93f8bd4a7462"
server_key_version: "encryptionkey@0"
I20260808 22:24:27.696640 31109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:24:27.728157 31109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:24:27.731632 31109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:24:27.731837 31109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:24:27.732491 31109 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:24:27.733425 31109 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:24:27.733492 31109 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:24:27.733553 31109 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:24:27.733582 31109 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:24:27.743103 31109 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.193:34985
I20260808 22:24:27.743134 31310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.193:34985 every 8 connection(s)
I20260808 22:24:27.744030 31109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
Aug 08 22:24:27 dist-test-slave-ncp9 krb5kdc[30953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227867, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.193@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
I20260808 22:24:27.753731 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 31109
I20260808 22:24:27.753858 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-0/wal/instance
I20260808 22:24:27.754148 9923 external_mini_cluster.cc:1428] Setting key ef00043d80632ae4deaf1b75561695ca
I20260808 22:24:27.757318 31076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:27.745899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.193:59313 (local address 127.9.176.254:38597)
0808 22:24:27.746066 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0808 22:24:27.746076 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 22:24:27.747101 (+ 1025us) server_negotiation.cc:407] Connection header received
0808 22:24:27.747973 (+ 872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:24:27.747976 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:24:27.748026 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:24:27.748103 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:24:27.749356 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:27.749906 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:27.750994 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:27.751241 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:27.751316 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:27.753808 (+ 2492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:24:27.753812 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:24:27.753814 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:24:27.753842 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:24:27.755271 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:24:27.755851 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:24:27.755855 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:24:27.755857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:24:27.755952 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:24:27.756241 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:24:27.756244 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:24:27.756245 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:24:27.756460 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:24:27.756566 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0808 22:24:27.757042 (+ 476us) server_negotiation.cc:299] Negotiation successful
0808 22:24:27.757166 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 22:24:27.758113 31314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:27.746265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:38597 (local address 127.9.176.193:59313)
0808 22:24:27.746934 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:27.746992 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0808 22:24:27.747786 (+ 794us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:24:27.748260 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:24:27.748267 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:24:27.748645 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:24:27.749218 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:24:27.749234 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:24:27.750009 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:24:27.750011 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:24:27.750890 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:24:27.750899 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:24:27.751372 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:24:27.751375 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:24:27.751525 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:27.752097 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:24:27.752113 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:24:27.753635 (+ 1522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:24:27.755388 (+ 1753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:24:27.755395 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:24:27.755409 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:24:27.755754 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:24:27.756033 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:24:27.756035 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:24:27.756036 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:24:27.756163 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:24:27.756559 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:24:27.756564 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:24:27.756799 (+ 235us) client_negotiation.cc:769] Sending connection context
0808 22:24:27.756995 (+ 196us) client_negotiation.cc:240] Negotiation successful
0808 22:24:27.757263 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":108,"threads_started":1}
I20260808 22:24:27.759402 31311 heartbeater.cc:347] Connected to a master server at 127.9.176.254:38597
I20260808 22:24:27.759636 31311 heartbeater.cc:464] Registering TS with master...
I20260808 22:24:27.760094 31311 heartbeater.cc:511] Master 127.9.176.254:38597 requested a full tablet report, sending...
I20260808 22:24:27.761785 31014 ts_manager.cc:194] Registered new tserver with Master: c725f56e88ba4e649a6e854587c1da74 (127.9.176.193:34985)
I20260808 22:24:27.763393 31014 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.176.193@KRBTEST.COM'} at 127.9.176.193:59313
WARNING: no policy specified for kudu/127.9.176.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.176.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.176.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.194.keytab.
Entry for principal kudu/127.9.176.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.194.keytab.
I20260808 22:24:27.781597 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.194:0
--local_ip_for_outbound_sockets=127.9.176.194
--webserver_interface=127.9.176.194
--webserver_port=0
--tserver_master_addrs=127.9.176.254:38597
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.194.keytab
--principal=kudu/127.9.176.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:33087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:24:27.893865 31317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:24:27.894100 31317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:24:27.894143 31317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:24:27.898097 31317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:24:27.898149 31317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:24:27.898218 31317 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.194
I20260808 22:24:27.903363 31317 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:33087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.194.keytab
--principal=kudu/127.9.176.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.176.194
--webserver_port=0
--tserver_master_addrs=127.9.176.254:38597
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31317
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.194
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:24:27.904354 31317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:24:27.905422 31317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:24:27.914968 31323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:24:27.914928 31327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:24:27.915030 31317 server_base.cc:1034] running on GCE node
W20260808 22:24:27.914979 31324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:24:27.915552 31317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:24:27.916143 31317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:24:27.917295 31317 hybrid_clock.cc:648] HybridClock initialized: now 1786227867917272 us; error 34 us; skew 500 ppm
Aug 08 22:24:27 dist-test-slave-ncp9 krb5kdc[30953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227867, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:24:27.919878 31317 init.cc:377] Logged in from keytab as kudu/127.9.176.194@KRBTEST.COM (short username kudu)
I20260808 22:24:27.920692 31317 webserver.cc:460] Webserver started at http://127.9.176.194:44689/ using document root <none> and password file <none>
I20260808 22:24:27.921233 31317 fs_manager.cc:359] Metadata directory not provided
I20260808 22:24:27.921294 31317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:24:27.921455 31317 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:24:27.923521 31317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-1/data/instance:
uuid: "b9bdb3f4a3d84fb2b48be7e89cd5672f"
format_stamp: "Formatted at 2026-08-08 22:24:27 on dist-test-slave-ncp9"
server_key: "d2e6403be0a1e99d03f9585ac7abcfdd"
server_key_iv: "2fe1edf73c12e43a6bc9fe61d223548d"
server_key_version: "encryptionkey@0"
I20260808 22:24:27.924029 31317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-1/wal/instance:
uuid: "b9bdb3f4a3d84fb2b48be7e89cd5672f"
format_stamp: "Formatted at 2026-08-08 22:24:27 on dist-test-slave-ncp9"
server_key: "d2e6403be0a1e99d03f9585ac7abcfdd"
server_key_iv: "2fe1edf73c12e43a6bc9fe61d223548d"
server_key_version: "encryptionkey@0"
I20260808 22:24:27.927639 31317 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260808 22:24:27.930037 31339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:24:27.931209 31317 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 22:24:27.931308 31317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-1/wal
uuid: "b9bdb3f4a3d84fb2b48be7e89cd5672f"
format_stamp: "Formatted at 2026-08-08 22:24:27 on dist-test-slave-ncp9"
server_key: "d2e6403be0a1e99d03f9585ac7abcfdd"
server_key_iv: "2fe1edf73c12e43a6bc9fe61d223548d"
server_key_version: "encryptionkey@0"
I20260808 22:24:27.931414 31317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:24:27.950333 31317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:24:27.953459 31317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:24:27.953650 31317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:24:27.954250 31317 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:24:27.955174 31317 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:24:27.955225 31317 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:24:27.955265 31317 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:24:27.955281 31317 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:24:27.964219 31317 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.194:37751
I20260808 22:24:27.964257 31499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.194:37751 every 8 connection(s)
I20260808 22:24:27.965173 31317 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
I20260808 22:24:27.967562 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 31317
I20260808 22:24:27.967684 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-1/wal/instance
I20260808 22:24:27.967954 9923 external_mini_cluster.cc:1428] Setting key f8cc6a11ca8bc3b729d37270ed81e5f7
Aug 08 22:24:27 dist-test-slave-ncp9 krb5kdc[30953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227867, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.194@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.176.195@KRBTEST.COM; defaulting to no policy
I20260808 22:24:27.979705 31076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:27.966952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.194:35103 (local address 127.9.176.254:38597)
0808 22:24:27.967083 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0808 22:24:27.967087 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:24:27.968255 (+ 1168us) server_negotiation.cc:407] Connection header received
0808 22:24:27.969264 (+ 1009us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:24:27.969267 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:24:27.969320 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:24:27.969379 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:24:27.970589 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:27.971310 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:27.973075 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:27.973296 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:27.973370 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:27.976220 (+ 2850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:24:27.976223 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:24:27.976225 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:24:27.976249 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:24:27.977572 (+ 1323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:24:27.978053 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:24:27.978057 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:24:27.978059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:24:27.978137 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:24:27.978486 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:24:27.978490 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:24:27.978492 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:24:27.978774 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:24:27.978898 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0808 22:24:27.979412 (+ 514us) server_negotiation.cc:299] Negotiation successful
0808 22:24:27.979577 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 22:24:27.980455 31505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:27.967325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:38597 (local address 127.9.176.194:35103)
0808 22:24:27.968068 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:27.968125 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0808 22:24:27.969027 (+ 902us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:24:27.969575 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:24:27.969583 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:24:27.969943 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:24:27.970435 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:24:27.970449 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:24:27.971477 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:24:27.971482 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:24:27.972942 (+ 1460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:24:27.972956 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:24:27.973451 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:24:27.973456 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:24:27.973662 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:27.974315 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:24:27.974342 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:24:27.976062 (+ 1720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:24:27.977682 (+ 1620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:24:27.977688 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:24:27.977704 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:24:27.977958 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:24:27.978220 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:24:27.978223 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:24:27.978225 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:24:27.978364 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:24:27.978892 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:24:27.978897 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:24:27.979154 (+ 257us) client_negotiation.cc:769] Sending connection context
0808 22:24:27.979314 (+ 160us) client_negotiation.cc:240] Negotiation successful
0808 22:24:27.979586 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":127,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.176.195@KRBTEST.COM" created.
I20260808 22:24:27.981726 31500 heartbeater.cc:347] Connected to a master server at 127.9.176.254:38597
I20260808 22:24:27.981998 31500 heartbeater.cc:464] Registering TS with master...
I20260808 22:24:27.982612 31500 heartbeater.cc:511] Master 127.9.176.254:38597 requested a full tablet report, sending...
I20260808 22:24:27.983824 31014 ts_manager.cc:194] Registered new tserver with Master: b9bdb3f4a3d84fb2b48be7e89cd5672f (127.9.176.194:37751)
I20260808 22:24:27.984615 31014 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.176.194@KRBTEST.COM'} at 127.9.176.194:35103
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.176.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.195.keytab.
Entry for principal kudu/127.9.176.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.195.keytab.
I20260808 22:24:27.995682 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.195:0
--local_ip_for_outbound_sockets=127.9.176.195
--webserver_interface=127.9.176.195
--webserver_port=0
--tserver_master_addrs=127.9.176.254:38597
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.195.keytab
--principal=kudu/127.9.176.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:33087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:24:28.105748 31511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:24:28.105993 31511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:24:28.106033 31511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:24:28.109902 31511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:24:28.109959 31511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:24:28.110014 31511 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.195
I20260808 22:24:28.115054 31511 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:33087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.195.keytab
--principal=kudu/127.9.176.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.176.195
--webserver_port=0
--tserver_master_addrs=127.9.176.254:38597
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31511
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.195
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:24:28.115993 31511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:24:28.116953 31511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:24:28.126269 31518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:24:28.126324 31511 server_base.cc:1034] running on GCE node
W20260808 22:24:28.126336 31519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:24:28.126261 31522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:24:28.126955 31511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:24:28.127542 31511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:24:28.128693 31511 hybrid_clock.cc:648] HybridClock initialized: now 1786227868128675 us; error 28 us; skew 500 ppm
Aug 08 22:24:28 dist-test-slave-ncp9 krb5kdc[30953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227868, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:24:28.131590 31511 init.cc:377] Logged in from keytab as kudu/127.9.176.195@KRBTEST.COM (short username kudu)
I20260808 22:24:28.132387 31511 webserver.cc:460] Webserver started at http://127.9.176.195:39893/ using document root <none> and password file <none>
I20260808 22:24:28.132937 31511 fs_manager.cc:359] Metadata directory not provided
I20260808 22:24:28.133008 31511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:24:28.133206 31511 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:24:28.135345 31511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-2/data/instance:
uuid: "e7519d2ff3a643fdbc922255349cc2d1"
format_stamp: "Formatted at 2026-08-08 22:24:28 on dist-test-slave-ncp9"
server_key: "ac780607a96c5bd1326617b7c3a4b343"
server_key_iv: "c95496a88a66d3ffaa5170a090833e43"
server_key_version: "encryptionkey@0"
I20260808 22:24:28.135950 31511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-2/wal/instance:
uuid: "e7519d2ff3a643fdbc922255349cc2d1"
format_stamp: "Formatted at 2026-08-08 22:24:28 on dist-test-slave-ncp9"
server_key: "ac780607a96c5bd1326617b7c3a4b343"
server_key_iv: "c95496a88a66d3ffaa5170a090833e43"
server_key_version: "encryptionkey@0"
I20260808 22:24:28.139456 31511 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260808 22:24:28.141722 31532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:24:28.142839 31511 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260808 22:24:28.142942 31511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-2/wal
uuid: "e7519d2ff3a643fdbc922255349cc2d1"
format_stamp: "Formatted at 2026-08-08 22:24:28 on dist-test-slave-ncp9"
server_key: "ac780607a96c5bd1326617b7c3a4b343"
server_key_iv: "c95496a88a66d3ffaa5170a090833e43"
server_key_version: "encryptionkey@0"
I20260808 22:24:28.143028 31511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:24:28.165375 31511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:24:28.167910 31511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:24:28.168076 31511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:24:28.168614 31511 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:24:28.169441 31511 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:24:28.169488 31511 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:24:28.169539 31511 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:24:28.169566 31511 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:24:28.179389 31511 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.195:44949
I20260808 22:24:28.179412 31702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.195:44949 every 8 connection(s)
I20260808 22:24:28.180207 31511 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
I20260808 22:24:28.181912 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 31511
I20260808 22:24:28.182029 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-2/wal/instance
I20260808 22:24:28.182345 9923 external_mini_cluster.cc:1428] Setting key 86522c2d834671fb184c3d9de98e9969
Aug 08 22:24:28 dist-test-slave-ncp9 krb5kdc[30953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227868, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.195@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
I20260808 22:24:28.192813 31076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:28.181880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.195:42157 (local address 127.9.176.254:38597)
0808 22:24:28.182012 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0808 22:24:28.182015 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:24:28.183313 (+ 1298us) server_negotiation.cc:407] Connection header received
0808 22:24:28.184409 (+ 1096us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:24:28.184413 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:24:28.184467 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:24:28.184539 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:24:28.185801 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:28.186441 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:28.187566 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:28.187774 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:28.187842 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:28.190074 (+ 2232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:24:28.190077 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:24:28.190079 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:24:28.190100 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:24:28.191247 (+ 1147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:24:28.191660 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:24:28.191662 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:24:28.191664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:24:28.191732 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:24:28.191985 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:24:28.191987 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:24:28.191988 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:24:28.192139 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:24:28.192218 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0808 22:24:28.192610 (+ 392us) server_negotiation.cc:299] Negotiation successful
0808 22:24:28.192700 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 22:24:28.193619 31708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:28.182389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:38597 (local address 127.9.176.195:42157)
0808 22:24:28.183155 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:28.183211 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0808 22:24:28.184182 (+ 971us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:24:28.184734 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:24:28.184741 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:24:28.185158 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:24:28.185670 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:24:28.185684 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:24:28.186554 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:24:28.186557 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:24:28.187482 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:24:28.187492 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:24:28.187875 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:24:28.187878 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:24:28.188015 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:28.188546 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:24:28.188562 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:24:28.189945 (+ 1383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:24:28.191338 (+ 1393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:24:28.191343 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:24:28.191354 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:24:28.191587 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:24:28.191795 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:24:28.191797 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:24:28.191798 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:24:28.191919 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:24:28.192219 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:24:28.192223 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:24:28.192433 (+ 210us) client_negotiation.cc:769] Sending connection context
0808 22:24:28.192578 (+ 145us) client_negotiation.cc:240] Negotiation successful
0808 22:24:28.192809 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":128,"threads_started":1}
I20260808 22:24:28.194566 31704 heartbeater.cc:347] Connected to a master server at 127.9.176.254:38597
I20260808 22:24:28.194808 31704 heartbeater.cc:464] Registering TS with master...
I20260808 22:24:28.195318 31704 heartbeater.cc:511] Master 127.9.176.254:38597 requested a full tablet report, sending...
I20260808 22:24:28.196306 31014 ts_manager.cc:194] Registered new tserver with Master: e7519d2ff3a643fdbc922255349cc2d1 (127.9.176.195:44949)
I20260808 22:24:28.196843 31014 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.176.195@KRBTEST.COM'} at 127.9.176.195:42157
I20260808 22:24:28.207422 9923 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 22:24:28.215463 31076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:28.209254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33434 (local address 127.9.176.254:38597)
0808 22:24:28.209364 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0808 22:24:28.209367 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:24:28.209519 (+ 152us) server_negotiation.cc:407] Connection header received
0808 22:24:28.209623 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:24:28.209626 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:24:28.209662 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:24:28.209715 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:24:28.210310 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:28.210792 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:28.211542 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:28.211738 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:28.211801 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:28.213009 (+ 1208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:24:28.213012 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:24:28.213014 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:24:28.213034 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:24:28.214257 (+ 1223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:24:28.214582 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:24:28.214584 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:24:28.214585 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:24:28.214655 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:24:28.214841 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:24:28.214843 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:24:28.214844 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:24:28.215029 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:24:28.215136 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0808 22:24:28.215273 (+ 137us) server_negotiation.cc:299] Negotiation successful
0808 22:24:28.215358 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260808 22:24:28.217952 31014 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33434:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 22:24:28.220829 31014 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 22:24:28.234303 31723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:28.228751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.195:44949 (local address 127.0.0.1:57838)
0808 22:24:28.229673 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:28.229684 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 22:24:28.229774 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:24:28.230482 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:24:28.230489 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:24:28.230514 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:24:28.230655 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:24:28.230662 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:24:28.231653 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:24:28.231656 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:24:28.232806 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:24:28.232815 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:24:28.233761 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:24:28.233764 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:24:28.233865 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:28.233908 (+ 43us) client_negotiation.cc:769] Sending connection context
0808 22:24:28.234010 (+ 102us) client_negotiation.cc:240] Negotiation successful
0808 22:24:28.234122 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":856,"spinlock_wait_cycles":2304,"thread_start_us":92,"threads_started":1}
I20260808 22:24:28.234335 31730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:28.229450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57838 (local address 127.9.176.195:44949)
0808 22:24:28.230052 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0808 22:24:28.230063 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 22:24:28.230082 (+ 19us) server_negotiation.cc:407] Connection header received
0808 22:24:28.230144 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:24:28.230149 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:24:28.230295 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:24:28.230455 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:24:28.230773 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:28.231509 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:28.232980 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:28.233644 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:28.233750 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:28.233892 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0808 22:24:28.234047 (+ 155us) server_negotiation.cc:299] Negotiation successful
0808 22:24:28.234146 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":180,"threads_started":1}
I20260808 22:24:28.235105 31725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:28.229253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.193:34985 (local address 127.0.0.1:55750)
0808 22:24:28.230109 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:28.230121 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 22:24:28.230215 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:24:28.230935 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:24:28.230937 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:24:28.230954 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:24:28.231031 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:24:28.231037 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:24:28.232185 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:24:28.232188 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:24:28.233378 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:24:28.233388 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:24:28.234676 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:24:28.234679 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:24:28.234765 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:28.234788 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 22:24:28.234852 (+ 64us) client_negotiation.cc:240] Negotiation successful
0808 22:24:28.234921 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":800,"mutex_wait_us":1,"thread_start_us":86,"threads_started":1}
I20260808 22:24:28.235119 31731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:28.229674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55750 (local address 127.9.176.193:34985)
0808 22:24:28.230515 (+ 841us) server_negotiation.cc:206] Beginning negotiation
0808 22:24:28.230530 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 22:24:28.230552 (+ 22us) server_negotiation.cc:407] Connection header received
0808 22:24:28.230623 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:24:28.230628 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:24:28.230818 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:24:28.230942 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:24:28.231144 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:28.232035 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:28.233505 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:28.234352 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:28.234486 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:28.234618 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0808 22:24:28.234897 (+ 279us) server_negotiation.cc:299] Negotiation successful
0808 22:24:28.234986 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":735,"thread_start_us":81,"threads_started":1}
I20260808 22:24:28.235855 31728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:28.228646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.194:37751 (local address 127.0.0.1:47462)
0808 22:24:28.229588 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:28.229636 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0808 22:24:28.229774 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:24:28.230556 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:24:28.230559 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:24:28.230578 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:24:28.230669 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:24:28.230674 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:24:28.232683 (+ 2009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:24:28.232686 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:24:28.233713 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:24:28.233723 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:24:28.235476 (+ 1753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:24:28.235480 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:24:28.235567 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:28.235590 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 22:24:28.235672 (+ 82us) client_negotiation.cc:240] Negotiation successful
0808 22:24:28.235734 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":776,"spinlock_wait_cycles":1408,"thread_start_us":84,"threads_started":1}
I20260808 22:24:28.236304 31727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:28.228800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47462 (local address 127.9.176.194:37751)
0808 22:24:28.229164 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0808 22:24:28.229178 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 22:24:28.229910 (+ 732us) server_negotiation.cc:407] Connection header received
0808 22:24:28.229996 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:24:28.230001 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:24:28.230222 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:24:28.230372 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:24:28.231667 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:28.232524 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:28.233838 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:28.234916 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:28.235011 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:28.235135 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0808 22:24:28.236069 (+ 934us) server_negotiation.cc:299] Negotiation successful
0808 22:24:28.236163 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":141,"threads_started":1}
I20260808 22:24:28.237795 31606 tablet_service.cc:1467] Processing CreateTablet for tablet 67d030e843364bf2aacc35e81a36d472 (DEFAULT_TABLE table=test-table [id=db5d12dae627402a90f58548a4da4260]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:24:28.238590 31409 tablet_service.cc:1467] Processing CreateTablet for tablet 67d030e843364bf2aacc35e81a36d472 (DEFAULT_TABLE table=test-table [id=db5d12dae627402a90f58548a4da4260]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:24:28.238941 31606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67d030e843364bf2aacc35e81a36d472. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:24:28.238555 31212 tablet_service.cc:1467] Processing CreateTablet for tablet 67d030e843364bf2aacc35e81a36d472 (DEFAULT_TABLE table=test-table [id=db5d12dae627402a90f58548a4da4260]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:24:28.239427 31409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67d030e843364bf2aacc35e81a36d472. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:24:28.239617 31212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67d030e843364bf2aacc35e81a36d472. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:24:28.245790 31732 tablet_bootstrap.cc:492] T 67d030e843364bf2aacc35e81a36d472 P b9bdb3f4a3d84fb2b48be7e89cd5672f: Bootstrap starting.
I20260808 22:24:28.247187 31733 tablet_bootstrap.cc:492] T 67d030e843364bf2aacc35e81a36d472 P c725f56e88ba4e649a6e854587c1da74: Bootstrap starting.
I20260808 22:24:28.247313 31734 tablet_bootstrap.cc:492] T 67d030e843364bf2aacc35e81a36d472 P e7519d2ff3a643fdbc922255349cc2d1: Bootstrap starting.
I20260808 22:24:28.248391 31732 tablet_bootstrap.cc:654] T 67d030e843364bf2aacc35e81a36d472 P b9bdb3f4a3d84fb2b48be7e89cd5672f: Neither blocks nor log segments found. Creating new log.
I20260808 22:24:28.249142 31732 log.cc:824] T 67d030e843364bf2aacc35e81a36d472 P b9bdb3f4a3d84fb2b48be7e89cd5672f: Log is configured to *not* fsync() on all Append() calls
I20260808 22:24:28.249361 31733 tablet_bootstrap.cc:654] T 67d030e843364bf2aacc35e81a36d472 P c725f56e88ba4e649a6e854587c1da74: Neither blocks nor log segments found. Creating new log.
I20260808 22:24:28.249763 31734 tablet_bootstrap.cc:654] T 67d030e843364bf2aacc35e81a36d472 P e7519d2ff3a643fdbc922255349cc2d1: Neither blocks nor log segments found. Creating new log.
I20260808 22:24:28.250558 31734 log.cc:824] T 67d030e843364bf2aacc35e81a36d472 P e7519d2ff3a643fdbc922255349cc2d1: Log is configured to *not* fsync() on all Append() calls
I20260808 22:24:28.250558 31733 log.cc:824] T 67d030e843364bf2aacc35e81a36d472 P c725f56e88ba4e649a6e854587c1da74: Log is configured to *not* fsync() on all Append() calls
I20260808 22:24:28.250855 31732 tablet_bootstrap.cc:492] T 67d030e843364bf2aacc35e81a36d472 P b9bdb3f4a3d84fb2b48be7e89cd5672f: No bootstrap required, opened a new log
I20260808 22:24:28.251037 31732 ts_tablet_manager.cc:1397] T 67d030e843364bf2aacc35e81a36d472 P b9bdb3f4a3d84fb2b48be7e89cd5672f: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260808 22:24:28.252372 31733 tablet_bootstrap.cc:492] T 67d030e843364bf2aacc35e81a36d472 P c725f56e88ba4e649a6e854587c1da74: No bootstrap required, opened a new log
I20260808 22:24:28.252483 31734 tablet_bootstrap.cc:492] T 67d030e843364bf2aacc35e81a36d472 P e7519d2ff3a643fdbc922255349cc2d1: No bootstrap required, opened a new log
I20260808 22:24:28.252588 31733 ts_tablet_manager.cc:1397] T 67d030e843364bf2aacc35e81a36d472 P c725f56e88ba4e649a6e854587c1da74: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260808 22:24:28.252672 31734 ts_tablet_manager.cc:1397] T 67d030e843364bf2aacc35e81a36d472 P e7519d2ff3a643fdbc922255349cc2d1: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260808 22:24:28.254844 31732 raft_consensus.cc:348] T 67d030e843364bf2aacc35e81a36d472 P b9bdb3f4a3d84fb2b48be7e89cd5672f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9bdb3f4a3d84fb2b48be7e89cd5672f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 37751 } } peers { permanent_uuid: "e7519d2ff3a643fdbc922255349cc2d1" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 44949 } } peers { permanent_uuid: "c725f56e88ba4e649a6e854587c1da74" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34985 } }
I20260808 22:24:28.255121 31732 raft_consensus.cc:374] T 67d030e843364bf2aacc35e81a36d472 P b9bdb3f4a3d84fb2b48be7e89cd5672f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:24:28.255219 31732 raft_consensus.cc:729] T 67d030e843364bf2aacc35e81a36d472 P b9bdb3f4a3d84fb2b48be7e89cd5672f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9bdb3f4a3d84fb2b48be7e89cd5672f, State: Initialized, Role: FOLLOWER
I20260808 22:24:28.255282 31733 raft_consensus.cc:348] T 67d030e843364bf2aacc35e81a36d472 P c725f56e88ba4e649a6e854587c1da74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9bdb3f4a3d84fb2b48be7e89cd5672f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 37751 } } peers { permanent_uuid: "e7519d2ff3a643fdbc922255349cc2d1" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 44949 } } peers { permanent_uuid: "c725f56e88ba4e649a6e854587c1da74" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34985 } }
I20260808 22:24:28.255517 31733 raft_consensus.cc:374] T 67d030e843364bf2aacc35e81a36d472 P c725f56e88ba4e649a6e854587c1da74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:24:28.255573 31733 raft_consensus.cc:729] T 67d030e843364bf2aacc35e81a36d472 P c725f56e88ba4e649a6e854587c1da74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c725f56e88ba4e649a6e854587c1da74, State: Initialized, Role: FOLLOWER
I20260808 22:24:28.255663 31732 consensus_queue.cc:260] T 67d030e843364bf2aacc35e81a36d472 P b9bdb3f4a3d84fb2b48be7e89cd5672f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9bdb3f4a3d84fb2b48be7e89cd5672f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 37751 } } peers { permanent_uuid: "e7519d2ff3a643fdbc922255349cc2d1" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 44949 } } peers { permanent_uuid: "c725f56e88ba4e649a6e854587c1da74" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34985 } }
I20260808 22:24:28.255960 31733 consensus_queue.cc:260] T 67d030e843364bf2aacc35e81a36d472 P c725f56e88ba4e649a6e854587c1da74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9bdb3f4a3d84fb2b48be7e89cd5672f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 37751 } } peers { permanent_uuid: "e7519d2ff3a643fdbc922255349cc2d1" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 44949 } } peers { permanent_uuid: "c725f56e88ba4e649a6e854587c1da74" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34985 } }
I20260808 22:24:28.256367 31500 heartbeater.cc:503] Master 127.9.176.254:38597 was elected leader, sending a full tablet report...
I20260808 22:24:28.256276 31734 raft_consensus.cc:348] T 67d030e843364bf2aacc35e81a36d472 P e7519d2ff3a643fdbc922255349cc2d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9bdb3f4a3d84fb2b48be7e89cd5672f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 37751 } } peers { permanent_uuid: "e7519d2ff3a643fdbc922255349cc2d1" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 44949 } } peers { permanent_uuid: "c725f56e88ba4e649a6e854587c1da74" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34985 } }
I20260808 22:24:28.256522 31734 raft_consensus.cc:374] T 67d030e843364bf2aacc35e81a36d472 P e7519d2ff3a643fdbc922255349cc2d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:24:28.256579 31734 raft_consensus.cc:729] T 67d030e843364bf2aacc35e81a36d472 P e7519d2ff3a643fdbc922255349cc2d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7519d2ff3a643fdbc922255349cc2d1, State: Initialized, Role: FOLLOWER
I20260808 22:24:28.256739 31732 ts_tablet_manager.cc:1428] T 67d030e843364bf2aacc35e81a36d472 P b9bdb3f4a3d84fb2b48be7e89cd5672f: Time spent starting tablet: real 0.006s user 0.000s sys 0.005s
I20260808 22:24:28.256995 31734 consensus_queue.cc:260] T 67d030e843364bf2aacc35e81a36d472 P e7519d2ff3a643fdbc922255349cc2d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9bdb3f4a3d84fb2b48be7e89cd5672f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 37751 } } peers { permanent_uuid: "e7519d2ff3a643fdbc922255349cc2d1" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 44949 } } peers { permanent_uuid: "c725f56e88ba4e649a6e854587c1da74" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34985 } }
I20260808 22:24:28.257571 31704 heartbeater.cc:503] Master 127.9.176.254:38597 was elected leader, sending a full tablet report...
I20260808 22:24:28.257612 31733 ts_tablet_manager.cc:1428] T 67d030e843364bf2aacc35e81a36d472 P c725f56e88ba4e649a6e854587c1da74: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260808 22:24:28.257920 31311 heartbeater.cc:503] Master 127.9.176.254:38597 was elected leader, sending a full tablet report...
I20260808 22:24:28.258760 31734 ts_tablet_manager.cc:1428] T 67d030e843364bf2aacc35e81a36d472 P e7519d2ff3a643fdbc922255349cc2d1: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260808 22:24:28.311419 31739 raft_consensus.cc:482] T 67d030e843364bf2aacc35e81a36d472 P c725f56e88ba4e649a6e854587c1da74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 22:24:28.311669 31739 raft_consensus.cc:504] T 67d030e843364bf2aacc35e81a36d472 P c725f56e88ba4e649a6e854587c1da74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9bdb3f4a3d84fb2b48be7e89cd5672f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 37751 } } peers { permanent_uuid: "e7519d2ff3a643fdbc922255349cc2d1" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 44949 } } peers { permanent_uuid: "c725f56e88ba4e649a6e854587c1da74" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34985 } }
I20260808 22:24:28.312778 31739 leader_election.cc:288] T 67d030e843364bf2aacc35e81a36d472 P c725f56e88ba4e649a6e854587c1da74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9bdb3f4a3d84fb2b48be7e89cd5672f (127.9.176.194:37751), e7519d2ff3a643fdbc922255349cc2d1 (127.9.176.195:44949)
I20260808 22:24:28.317935 31727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:28.313144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.193:52079 (local address 127.9.176.194:37751)
0808 22:24:28.313275 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0808 22:24:28.313280 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:24:28.313510 (+ 230us) server_negotiation.cc:407] Connection header received
0808 22:24:28.313666 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:24:28.313669 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:24:28.313723 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:24:28.313805 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:24:28.314162 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:28.314932 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:28.316666 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:28.317311 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:28.317430 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:28.317484 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0808 22:24:28.317733 (+ 249us) server_negotiation.cc:299] Negotiation successful
0808 22:24:28.317804 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 22:24:28.317935 31730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:28.313089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.193:50213 (local address 127.9.176.195:44949)
0808 22:24:28.313224 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0808 22:24:28.313229 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:24:28.313567 (+ 338us) server_negotiation.cc:407] Connection header received
0808 22:24:28.313679 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:24:28.313682 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:24:28.313727 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:24:28.313804 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:24:28.314147 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:28.314936 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:28.316666 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:28.317311 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:28.317430 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:28.317484 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0808 22:24:28.317729 (+ 245us) server_negotiation.cc:299] Negotiation successful
0808 22:24:28.317804 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 22:24:28.317988 31742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:28.313004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.194:37751 (local address 127.9.176.193:52079)
0808 22:24:28.313402 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:28.313418 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 22:24:28.313549 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:24:28.313821 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:24:28.313824 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:24:28.313855 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:24:28.314036 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:24:28.314044 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:24:28.315044 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:24:28.315048 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:24:28.316542 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:24:28.316555 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:24:28.317433 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:24:28.317437 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:24:28.317589 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:28.317629 (+ 40us) client_negotiation.cc:769] Sending connection context
0808 22:24:28.317700 (+ 71us) client_negotiation.cc:240] Negotiation successful
0808 22:24:28.317817 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"mutex_wait_us":232,"spinlock_wait_cycles":10368,"thread_start_us":112,"threads_started":1}
I20260808 22:24:28.317991 31744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:28.313004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.195:44949 (local address 127.9.176.193:50213)
0808 22:24:28.313490 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:28.313501 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 22:24:28.313582 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:24:28.313821 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:24:28.313824 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:24:28.313849 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:24:28.314042 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:24:28.314047 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:24:28.315044 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:24:28.315048 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:24:28.316542 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:24:28.316555 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:24:28.317433 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:24:28.317437 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:24:28.317561 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:28.317615 (+ 54us) client_negotiation.cc:769] Sending connection context
0808 22:24:28.317704 (+ 89us) client_negotiation.cc:240] Negotiation successful
0808 22:24:28.317810 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"mutex_wait_us":356,"spinlock_wait_cycles":2560,"thread_start_us":99,"threads_started":1}
I20260808 22:24:28.318696 31437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67d030e843364bf2aacc35e81a36d472" candidate_uuid: "c725f56e88ba4e649a6e854587c1da74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9bdb3f4a3d84fb2b48be7e89cd5672f" is_pre_election: true
I20260808 22:24:28.318698 31633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67d030e843364bf2aacc35e81a36d472" candidate_uuid: "c725f56e88ba4e649a6e854587c1da74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7519d2ff3a643fdbc922255349cc2d1" is_pre_election: true
I20260808 22:24:28.318972 31437 raft_consensus.cc:2454] T 67d030e843364bf2aacc35e81a36d472 P b9bdb3f4a3d84fb2b48be7e89cd5672f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c725f56e88ba4e649a6e854587c1da74 in term 0.
I20260808 22:24:28.318980 31633 raft_consensus.cc:2454] T 67d030e843364bf2aacc35e81a36d472 P e7519d2ff3a643fdbc922255349cc2d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c725f56e88ba4e649a6e854587c1da74 in term 0.
I20260808 22:24:28.319419 31138 leader_election.cc:302] T 67d030e843364bf2aacc35e81a36d472 P c725f56e88ba4e649a6e854587c1da74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c725f56e88ba4e649a6e854587c1da74, e7519d2ff3a643fdbc922255349cc2d1; no voters:
I20260808 22:24:28.319715 31739 raft_consensus.cc:2781] T 67d030e843364bf2aacc35e81a36d472 P c725f56e88ba4e649a6e854587c1da74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 22:24:28.319793 31739 raft_consensus.cc:482] T 67d030e843364bf2aacc35e81a36d472 P c725f56e88ba4e649a6e854587c1da74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 22:24:28.319840 31739 raft_consensus.cc:3037] T 67d030e843364bf2aacc35e81a36d472 P c725f56e88ba4e649a6e854587c1da74 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:24:28.320876 31739 raft_consensus.cc:504] T 67d030e843364bf2aacc35e81a36d472 P c725f56e88ba4e649a6e854587c1da74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9bdb3f4a3d84fb2b48be7e89cd5672f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 37751 } } peers { permanent_uuid: "e7519d2ff3a643fdbc922255349cc2d1" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 44949 } } peers { permanent_uuid: "c725f56e88ba4e649a6e854587c1da74" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34985 } }
I20260808 22:24:28.321237 31739 leader_election.cc:288] T 67d030e843364bf2aacc35e81a36d472 P c725f56e88ba4e649a6e854587c1da74 [CANDIDATE]: Term 1 election: Requested vote from peers b9bdb3f4a3d84fb2b48be7e89cd5672f (127.9.176.194:37751), e7519d2ff3a643fdbc922255349cc2d1 (127.9.176.195:44949)
I20260808 22:24:28.321570 31437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67d030e843364bf2aacc35e81a36d472" candidate_uuid: "c725f56e88ba4e649a6e854587c1da74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9bdb3f4a3d84fb2b48be7e89cd5672f"
I20260808 22:24:28.321570 31633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67d030e843364bf2aacc35e81a36d472" candidate_uuid: "c725f56e88ba4e649a6e854587c1da74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7519d2ff3a643fdbc922255349cc2d1"
I20260808 22:24:28.321687 31633 raft_consensus.cc:3037] T 67d030e843364bf2aacc35e81a36d472 P e7519d2ff3a643fdbc922255349cc2d1 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:24:28.321687 31437 raft_consensus.cc:3037] T 67d030e843364bf2aacc35e81a36d472 P b9bdb3f4a3d84fb2b48be7e89cd5672f [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:24:28.322971 31633 raft_consensus.cc:2454] T 67d030e843364bf2aacc35e81a36d472 P e7519d2ff3a643fdbc922255349cc2d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c725f56e88ba4e649a6e854587c1da74 in term 1.
I20260808 22:24:28.322973 31437 raft_consensus.cc:2454] T 67d030e843364bf2aacc35e81a36d472 P b9bdb3f4a3d84fb2b48be7e89cd5672f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c725f56e88ba4e649a6e854587c1da74 in term 1.
I20260808 22:24:28.323230 31138 leader_election.cc:302] T 67d030e843364bf2aacc35e81a36d472 P c725f56e88ba4e649a6e854587c1da74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c725f56e88ba4e649a6e854587c1da74, e7519d2ff3a643fdbc922255349cc2d1; no voters:
I20260808 22:24:28.323408 31739 raft_consensus.cc:2781] T 67d030e843364bf2aacc35e81a36d472 P c725f56e88ba4e649a6e854587c1da74 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:24:28.323589 31739 raft_consensus.cc:686] T 67d030e843364bf2aacc35e81a36d472 P c725f56e88ba4e649a6e854587c1da74 [term 1 LEADER]: Becoming Leader. State: Replica: c725f56e88ba4e649a6e854587c1da74, State: Running, Role: LEADER
I20260808 22:24:28.323882 31739 consensus_queue.cc:237] T 67d030e843364bf2aacc35e81a36d472 P c725f56e88ba4e649a6e854587c1da74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9bdb3f4a3d84fb2b48be7e89cd5672f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 37751 } } peers { permanent_uuid: "e7519d2ff3a643fdbc922255349cc2d1" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 44949 } } peers { permanent_uuid: "c725f56e88ba4e649a6e854587c1da74" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34985 } }
I20260808 22:24:28.327188 31009 catalog_manager.cc:5495] T 67d030e843364bf2aacc35e81a36d472 P c725f56e88ba4e649a6e854587c1da74 reported cstate change: term changed from 0 to 1, leader changed from <none> to c725f56e88ba4e649a6e854587c1da74 (127.9.176.193). New cstate: current_term: 1 leader_uuid: "c725f56e88ba4e649a6e854587c1da74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9bdb3f4a3d84fb2b48be7e89cd5672f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 37751 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7519d2ff3a643fdbc922255349cc2d1" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 44949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c725f56e88ba4e649a6e854587c1da74" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34985 } health_report { overall_health: HEALTHY } } }
I20260808 22:24:28.356232 31731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:28.353084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55766 (local address 127.9.176.193:34985)
0808 22:24:28.353219 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0808 22:24:28.353223 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:24:28.353238 (+ 15us) server_negotiation.cc:407] Connection header received
0808 22:24:28.353354 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:24:28.353356 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:24:28.353411 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:24:28.353499 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:24:28.353682 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:28.354189 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:28.355014 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:28.355221 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:28.355284 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:28.355503 (+ 219us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:24:28.355911 (+ 408us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:24:28.355994 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0808 22:24:28.356031 (+ 37us) server_negotiation.cc:299] Negotiation successful
0808 22:24:28.356080 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260808 22:24:28.361889 31437 raft_consensus.cc:1261] T 67d030e843364bf2aacc35e81a36d472 P b9bdb3f4a3d84fb2b48be7e89cd5672f [term 1 FOLLOWER]: Refusing update from remote peer c725f56e88ba4e649a6e854587c1da74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:24:28.361889 31633 raft_consensus.cc:1261] T 67d030e843364bf2aacc35e81a36d472 P e7519d2ff3a643fdbc922255349cc2d1 [term 1 FOLLOWER]: Refusing update from remote peer c725f56e88ba4e649a6e854587c1da74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:24:28.362582 31745 consensus_queue.cc:1035] T 67d030e843364bf2aacc35e81a36d472 P c725f56e88ba4e649a6e854587c1da74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9bdb3f4a3d84fb2b48be7e89cd5672f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 37751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:24:28.362756 31739 consensus_queue.cc:1035] T 67d030e843364bf2aacc35e81a36d472 P c725f56e88ba4e649a6e854587c1da74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7519d2ff3a643fdbc922255349cc2d1" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 44949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:24:28.370481 31750 mvcc.cc:205] Tried to move back new op lower bound from 7316389348803461120 to 7316389348657569792. Current Snapshot: MvccSnapshot[applied={T|T < 7316389348803461120}]
I20260808 22:24:28.379415 31012 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33434:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:24:28.379595 31012 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33434:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:24:28.381893 31012 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 35e08196664442e9b7f686512baa70f5: Sending DeleteTablet for 3 replicas of tablet 67d030e843364bf2aacc35e81a36d472
I20260808 22:24:28.382769 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 31109
I20260808 22:24:28.382754 31212 tablet_service.cc:1514] Processing DeleteTablet for tablet 67d030e843364bf2aacc35e81a36d472 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:24:28 UTC) from {username='kudu', principal='kudu/127.9.176.254@KRBTEST.COM'} at 127.0.0.1:55750
I20260808 22:24:28.382776 31606 tablet_service.cc:1514] Processing DeleteTablet for tablet 67d030e843364bf2aacc35e81a36d472 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:24:28 UTC) from {username='kudu', principal='kudu/127.9.176.254@KRBTEST.COM'} at 127.0.0.1:57838
I20260808 22:24:28.383215 31409 tablet_service.cc:1514] Processing DeleteTablet for tablet 67d030e843364bf2aacc35e81a36d472 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:24:28 UTC) from {username='kudu', principal='kudu/127.9.176.254@KRBTEST.COM'} at 127.0.0.1:47462
I20260808 22:24:28.391548 31766 tablet_replica.cc:330] T 67d030e843364bf2aacc35e81a36d472 P e7519d2ff3a643fdbc922255349cc2d1: stopping tablet replica
I20260808 22:24:28.391909 31766 raft_consensus.cc:2229] T 67d030e843364bf2aacc35e81a36d472 P e7519d2ff3a643fdbc922255349cc2d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:24:28.391983 31768 tablet_replica.cc:330] T 67d030e843364bf2aacc35e81a36d472 P b9bdb3f4a3d84fb2b48be7e89cd5672f: stopping tablet replica
I20260808 22:24:28.392262 31766 raft_consensus.cc:2258] T 67d030e843364bf2aacc35e81a36d472 P e7519d2ff3a643fdbc922255349cc2d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:24:28.392328 31768 raft_consensus.cc:2229] T 67d030e843364bf2aacc35e81a36d472 P b9bdb3f4a3d84fb2b48be7e89cd5672f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:24:28.392616 31768 raft_consensus.cc:2258] T 67d030e843364bf2aacc35e81a36d472 P b9bdb3f4a3d84fb2b48be7e89cd5672f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:24:28.393934 31766 ts_tablet_manager.cc:1905] T 67d030e843364bf2aacc35e81a36d472 P e7519d2ff3a643fdbc922255349cc2d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:24:28.393934 31768 ts_tablet_manager.cc:1905] T 67d030e843364bf2aacc35e81a36d472 P b9bdb3f4a3d84fb2b48be7e89cd5672f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:24:28.396662 31766 ts_tablet_manager.cc:1918] T 67d030e843364bf2aacc35e81a36d472 P e7519d2ff3a643fdbc922255349cc2d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 22:24:28.396661 31768 ts_tablet_manager.cc:1918] T 67d030e843364bf2aacc35e81a36d472 P b9bdb3f4a3d84fb2b48be7e89cd5672f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 22:24:28.396770 31766 log.cc:1196] T 67d030e843364bf2aacc35e81a36d472 P e7519d2ff3a643fdbc922255349cc2d1: Deleting WAL directory at /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-2/wal/wals/67d030e843364bf2aacc35e81a36d472
I20260808 22:24:28.396770 31768 log.cc:1196] T 67d030e843364bf2aacc35e81a36d472 P b9bdb3f4a3d84fb2b48be7e89cd5672f: Deleting WAL directory at /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-1/wal/wals/67d030e843364bf2aacc35e81a36d472
I20260808 22:24:28.397145 31768 ts_tablet_manager.cc:1939] T 67d030e843364bf2aacc35e81a36d472 P b9bdb3f4a3d84fb2b48be7e89cd5672f: Deleting consensus metadata
I20260808 22:24:28.397145 31766 ts_tablet_manager.cc:1939] T 67d030e843364bf2aacc35e81a36d472 P e7519d2ff3a643fdbc922255349cc2d1: Deleting consensus metadata
W20260808 22:24:28.398175 30999 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.176.193:34985 (error 108)
I20260808 22:24:28.398463 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 31317
I20260808 22:24:28.398440 30996 catalog_manager.cc:4841] TS b9bdb3f4a3d84fb2b48be7e89cd5672f (127.9.176.194:37751): tablet 67d030e843364bf2aacc35e81a36d472 (table test-table [id=db5d12dae627402a90f58548a4da4260]) successfully deleted
I20260808 22:24:28.398674 31000 catalog_manager.cc:4841] TS e7519d2ff3a643fdbc922255349cc2d1 (127.9.176.195:44949): tablet 67d030e843364bf2aacc35e81a36d472 (table test-table [id=db5d12dae627402a90f58548a4da4260]) successfully deleted
I20260808 22:24:28.401563 31728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:28.401254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.193:34985 (local address 127.0.0.1:55778)
0808 22:24:28.401394 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:28.401474 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.193:34985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260808 22:24:28.401822 30999 catalog_manager.cc:4584] TS c725f56e88ba4e649a6e854587c1da74 (127.9.176.193:34985): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 67d030e843364bf2aacc35e81a36d472: Network error: Client connection negotiation failed: client connection to 127.9.176.193:34985: connect: Connection refused (error 111)
I20260808 22:24:28.407835 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 31511
I20260808 22:24:28.424770 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 30971
I20260808 22:24:28.438684 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:38597
--webserver_interface=127.9.176.254
--webserver_port=40381
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab
--principal=kudu/127.9.176.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:33087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.176.254:38597
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:24:28.548166 31770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:24:28.548436 31770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:24:28.548476 31770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:24:28.552454 31770 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:24:28.552512 31770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:24:28.552531 31770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:24:28.552551 31770 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:24:28.552568 31770 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:24:28.557801 31770 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:33087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.9.176.254:38597
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab
--principal=kudu/127.9.176.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:38597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.176.254
--webserver_port=40381
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31770
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:24:28.558825 31770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:24:28.559830 31770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:24:28.574664 31786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:24:28.574729 31770 server_base.cc:1034] running on GCE node
W20260808 22:24:28.574661 31792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:24:28.574738 31790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:24:28.575523 31770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:24:28.576627 31770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:24:28.577781 31770 hybrid_clock.cc:648] HybridClock initialized: now 1786227868577749 us; error 46 us; skew 500 ppm
Aug 08 22:24:28 dist-test-slave-ncp9 krb5kdc[30953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227868, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:24:28.580410 31770 init.cc:377] Logged in from keytab as kudu/127.9.176.254@KRBTEST.COM (short username kudu)
I20260808 22:24:28.581336 31770 webserver.cc:460] Webserver started at http://127.9.176.254:40381/ using document root <none> and password file <none>
I20260808 22:24:28.581871 31770 fs_manager.cc:359] Metadata directory not provided
I20260808 22:24:28.581954 31770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:24:28.585779 31770 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260808 22:24:28.587815 31802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:24:28.589043 31770 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 22:24:28.589167 31770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/master-0/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/master-0/wal
uuid: "35e08196664442e9b7f686512baa70f5"
format_stamp: "Formatted at 2026-08-08 22:24:27 on dist-test-slave-ncp9"
server_key: "7653e1e631037b8f7bd49e7fad710b74"
server_key_iv: "86cf734e2347e21a839717c26f3d69a7"
server_key_version: "encryptionkey@0"
I20260808 22:24:28.589747 31770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:24:28.618880 31770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:24:28.621980 31770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:24:28.622172 31770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:24:28.629663 31770 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.254:38597
I20260808 22:24:28.629968 31871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.254:38597 every 8 connection(s)
I20260808 22:24:28.630846 31770 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
I20260808 22:24:28.635329 31872 sys_catalog.cc:263] Verifying existing consensus state
I20260808 22:24:28.635316 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 31770
I20260808 22:24:28.635804 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.193:34985
--local_ip_for_outbound_sockets=127.9.176.193
--tserver_master_addrs=127.9.176.254:38597
--webserver_port=42243
--webserver_interface=127.9.176.193
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.193.keytab
--principal=kudu/127.9.176.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:33087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 22:24:28.637499 31872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35e08196664442e9b7f686512baa70f5: Bootstrap starting.
I20260808 22:24:28.655187 31872 log.cc:824] T 00000000000000000000000000000000 P 35e08196664442e9b7f686512baa70f5: Log is configured to *not* fsync() on all Append() calls
I20260808 22:24:28.663991 31872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35e08196664442e9b7f686512baa70f5: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 22:24:28.664338 31872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35e08196664442e9b7f686512baa70f5: Bootstrap complete.
I20260808 22:24:28.668331 31872 raft_consensus.cc:348] T 00000000000000000000000000000000 P 35e08196664442e9b7f686512baa70f5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35e08196664442e9b7f686512baa70f5" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 38597 } }
I20260808 22:24:28.668741 31872 raft_consensus.cc:729] T 00000000000000000000000000000000 P 35e08196664442e9b7f686512baa70f5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35e08196664442e9b7f686512baa70f5, State: Initialized, Role: FOLLOWER
I20260808 22:24:28.669266 31872 consensus_queue.cc:260] T 00000000000000000000000000000000 P 35e08196664442e9b7f686512baa70f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35e08196664442e9b7f686512baa70f5" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 38597 } }
I20260808 22:24:28.669392 31872 raft_consensus.cc:388] T 00000000000000000000000000000000 P 35e08196664442e9b7f686512baa70f5 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:24:28.669469 31872 raft_consensus.cc:482] T 00000000000000000000000000000000 P 35e08196664442e9b7f686512baa70f5 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:24:28.669559 31872 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 35e08196664442e9b7f686512baa70f5 [term 1 FOLLOWER]: Advancing to term 2
I20260808 22:24:28.671455 31872 raft_consensus.cc:504] T 00000000000000000000000000000000 P 35e08196664442e9b7f686512baa70f5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35e08196664442e9b7f686512baa70f5" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 38597 } }
I20260808 22:24:28.671764 31872 leader_election.cc:302] T 00000000000000000000000000000000 P 35e08196664442e9b7f686512baa70f5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 35e08196664442e9b7f686512baa70f5; no voters:
I20260808 22:24:28.672070 31872 leader_election.cc:288] T 00000000000000000000000000000000 P 35e08196664442e9b7f686512baa70f5 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260808 22:24:28.672169 31876 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 35e08196664442e9b7f686512baa70f5 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 22:24:28.672374 31876 raft_consensus.cc:686] T 00000000000000000000000000000000 P 35e08196664442e9b7f686512baa70f5 [term 2 LEADER]: Becoming Leader. State: Replica: 35e08196664442e9b7f686512baa70f5, State: Running, Role: LEADER
I20260808 22:24:28.672721 31876 consensus_queue.cc:237] T 00000000000000000000000000000000 P 35e08196664442e9b7f686512baa70f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35e08196664442e9b7f686512baa70f5" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 38597 } }
I20260808 22:24:28.673322 31872 sys_catalog.cc:564] T 00000000000000000000000000000000 P 35e08196664442e9b7f686512baa70f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:24:28.673694 31876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35e08196664442e9b7f686512baa70f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 35e08196664442e9b7f686512baa70f5. Latest consensus state: current_term: 2 leader_uuid: "35e08196664442e9b7f686512baa70f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35e08196664442e9b7f686512baa70f5" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 38597 } } }
I20260808 22:24:28.673812 31876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35e08196664442e9b7f686512baa70f5 [sys.catalog]: This master's current role is: LEADER
I20260808 22:24:28.674919 31877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35e08196664442e9b7f686512baa70f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "35e08196664442e9b7f686512baa70f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35e08196664442e9b7f686512baa70f5" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 38597 } } }
I20260808 22:24:28.675079 31877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35e08196664442e9b7f686512baa70f5 [sys.catalog]: This master's current role is: LEADER
I20260808 22:24:28.675282 31880 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:24:28.679752 31880 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:24:28.680370 31880 catalog_manager.cc:1259] Loaded cluster ID: 9f79495c212b4ff4a668128b2376b2af
I20260808 22:24:28.680433 31880 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:24:28.681994 31880 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260808 22:24:28.686743 31880 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260808 22:24:28.692086 31880 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:24:28.692840 31880 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 35e08196664442e9b7f686512baa70f5: Loaded TSK: 0
I20260808 22:24:28.694095 31880 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 22:24:28.749469 31874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:24:28.749717 31874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:24:28.749776 31874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:24:28.753684 31874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:24:28.753737 31874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:24:28.753809 31874 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.193
I20260808 22:24:28.758882 31874 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:33087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.193.keytab
--principal=kudu/127.9.176.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.193:34985
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.176.193
--webserver_port=42243
--tserver_master_addrs=127.9.176.254:38597
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31874
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.193
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:24:28.759836 31874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:24:28.760790 31874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:24:28.771050 31906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:24:28.771057 31905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:24:28.771431 31909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:24:28.771559 31874 server_base.cc:1034] running on GCE node
I20260808 22:24:28.771919 31874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:24:28.772533 31874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:24:28.773697 31874 hybrid_clock.cc:648] HybridClock initialized: now 1786227868773617 us; error 96 us; skew 500 ppm
Aug 08 22:24:28 dist-test-slave-ncp9 krb5kdc[30953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227868, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:24:28.776543 31874 init.cc:377] Logged in from keytab as kudu/127.9.176.193@KRBTEST.COM (short username kudu)
I20260808 22:24:28.777453 31874 webserver.cc:460] Webserver started at http://127.9.176.193:42243/ using document root <none> and password file <none>
I20260808 22:24:28.777972 31874 fs_manager.cc:359] Metadata directory not provided
I20260808 22:24:28.778059 31874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:24:28.781733 31874 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260808 22:24:28.783733 31919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:24:28.784885 31874 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 22:24:28.785023 31874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-0/wal
uuid: "c725f56e88ba4e649a6e854587c1da74"
format_stamp: "Formatted at 2026-08-08 22:24:27 on dist-test-slave-ncp9"
server_key: "c52a2e17aa4900cef485315f7c3cbfe0"
server_key_iv: "abb258a6a786adb289fd93f8bd4a7462"
server_key_version: "encryptionkey@0"
I20260808 22:24:28.785539 31874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:24:28.806713 31874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:24:28.809670 31874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:24:28.809837 31874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:24:28.810420 31874 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:24:28.811529 31928 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260808 22:24:28.815526 31874 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260808 22:24:28.815577 31874 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260808 22:24:28.815624 31874 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260808 22:24:28.817215 31874 ts_tablet_manager.cc:610] Registered 1 tablets
I20260808 22:24:28.817270 31874 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260808 22:24:28.817548 31928 tablet_bootstrap.cc:492] T 67d030e843364bf2aacc35e81a36d472 P c725f56e88ba4e649a6e854587c1da74: Bootstrap starting.
I20260808 22:24:28.827991 31874 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.193:34985
I20260808 22:24:28.828030 32084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.193:34985 every 8 connection(s)
I20260808 22:24:28.829185 31874 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
I20260808 22:24:28.833210 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 31874
I20260808 22:24:28.833706 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.194:37751
--local_ip_for_outbound_sockets=127.9.176.194
--tserver_master_addrs=127.9.176.254:38597
--webserver_port=44689
--webserver_interface=127.9.176.194
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.194.keytab
--principal=kudu/127.9.176.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:33087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 22:24:28.839311 31928 log.cc:824] T 67d030e843364bf2aacc35e81a36d472 P c725f56e88ba4e649a6e854587c1da74: Log is configured to *not* fsync() on all Append() calls
Aug 08 22:24:28 dist-test-slave-ncp9 krb5kdc[30953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227868, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.193@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
I20260808 22:24:28.846325 32089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:28.831288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.193:46985 (local address 127.9.176.254:38597)
0808 22:24:28.831660 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0808 22:24:28.831687 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0808 22:24:28.832016 (+ 329us) server_negotiation.cc:407] Connection header received
0808 22:24:28.833110 (+ 1094us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:24:28.833121 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:24:28.833455 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:24:28.833756 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:24:28.835030 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:28.836401 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:28.838254 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:28.839006 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:28.839191 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:28.841713 (+ 2522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:24:28.841719 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:24:28.841730 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:24:28.841759 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:24:28.843314 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:24:28.843804 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:24:28.843809 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:24:28.843813 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:24:28.843929 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:24:28.844248 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:24:28.844250 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:24:28.844252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:24:28.844580 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:24:28.844696 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0808 22:24:28.845256 (+ 560us) server_negotiation.cc:299] Negotiation successful
0808 22:24:28.845508 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":99,"threads_started":1}
I20260808 22:24:28.846338 32090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:28.831349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:38597 (local address 127.9.176.193:46985)
0808 22:24:28.831852 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:28.831895 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 22:24:28.832686 (+ 791us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:24:28.834043 (+ 1357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:24:28.834052 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:24:28.834403 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:24:28.834863 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:24:28.834877 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:24:28.836603 (+ 1726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:24:28.836607 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:24:28.838116 (+ 1509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:24:28.838129 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:24:28.839154 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:24:28.839158 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:24:28.839379 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:28.840035 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:24:28.840055 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:24:28.841534 (+ 1479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:24:28.843427 (+ 1893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:24:28.843433 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:24:28.843446 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:24:28.843700 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:24:28.844026 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:24:28.844029 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:24:28.844031 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:24:28.844162 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:24:28.844690 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:24:28.844696 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:24:28.844961 (+ 265us) client_negotiation.cc:769] Sending connection context
0808 22:24:28.845126 (+ 165us) client_negotiation.cc:240] Negotiation successful
0808 22:24:28.845444 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":150,"threads_started":1}
I20260808 22:24:28.848193 31928 tablet_bootstrap.cc:492] T 67d030e843364bf2aacc35e81a36d472 P c725f56e88ba4e649a6e854587c1da74: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 22:24:28.848606 31928 tablet_bootstrap.cc:492] T 67d030e843364bf2aacc35e81a36d472 P c725f56e88ba4e649a6e854587c1da74: Bootstrap complete.
I20260808 22:24:28.848973 32086 heartbeater.cc:347] Connected to a master server at 127.9.176.254:38597
I20260808 22:24:28.849186 32086 heartbeater.cc:464] Registering TS with master...
I20260808 22:24:28.849654 32086 heartbeater.cc:511] Master 127.9.176.254:38597 requested a full tablet report, sending...
I20260808 22:24:28.850114 31928 ts_tablet_manager.cc:1397] T 67d030e843364bf2aacc35e81a36d472 P c725f56e88ba4e649a6e854587c1da74: Time spent bootstrapping tablet: real 0.033s user 0.019s sys 0.012s
I20260808 22:24:28.851843 31823 ts_manager.cc:194] Registered new tserver with Master: c725f56e88ba4e649a6e854587c1da74 (127.9.176.193:34985)
I20260808 22:24:28.852978 31928 raft_consensus.cc:348] T 67d030e843364bf2aacc35e81a36d472 P c725f56e88ba4e649a6e854587c1da74 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9bdb3f4a3d84fb2b48be7e89cd5672f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 37751 } } peers { permanent_uuid: "e7519d2ff3a643fdbc922255349cc2d1" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 44949 } } peers { permanent_uuid: "c725f56e88ba4e649a6e854587c1da74" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34985 } }
I20260808 22:24:28.853529 31928 raft_consensus.cc:729] T 67d030e843364bf2aacc35e81a36d472 P c725f56e88ba4e649a6e854587c1da74 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c725f56e88ba4e649a6e854587c1da74, State: Initialized, Role: FOLLOWER
I20260808 22:24:28.854017 31928 consensus_queue.cc:260] T 67d030e843364bf2aacc35e81a36d472 P c725f56e88ba4e649a6e854587c1da74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9bdb3f4a3d84fb2b48be7e89cd5672f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 37751 } } peers { permanent_uuid: "e7519d2ff3a643fdbc922255349cc2d1" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 44949 } } peers { permanent_uuid: "c725f56e88ba4e649a6e854587c1da74" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34985 } }
I20260808 22:24:28.854987 31928 ts_tablet_manager.cc:1428] T 67d030e843364bf2aacc35e81a36d472 P c725f56e88ba4e649a6e854587c1da74: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260808 22:24:28.855270 31823 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.176.193@KRBTEST.COM'} at 127.9.176.193:46985
I20260808 22:24:28.857501 32086 heartbeater.cc:503] Master 127.9.176.254:38597 was elected leader, sending a full tablet report...
Aug 08 22:24:28 dist-test-slave-ncp9 krb5kdc[30953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227868, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.254@KRBTEST.COM for kudu/127.9.176.193@KRBTEST.COM
I20260808 22:24:28.866292 32096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:28.854620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.193:34985 (local address 127.0.0.1:48524)
0808 22:24:28.855622 (+ 1002us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:28.855672 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0808 22:24:28.855804 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:24:28.856304 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:24:28.856310 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:24:28.856731 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:24:28.856854 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:24:28.856861 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:24:28.859017 (+ 2156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:24:28.859021 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:24:28.860005 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:24:28.860018 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:24:28.860559 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:24:28.860562 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:24:28.860671 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:28.861215 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:24:28.861229 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:24:28.862633 (+ 1404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:24:28.864669 (+ 2036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:24:28.864678 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:24:28.864682 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:24:28.864915 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:24:28.865244 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:24:28.865247 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:24:28.865248 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:24:28.865336 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:24:28.865808 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:24:28.865813 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:24:28.865910 (+ 97us) client_negotiation.cc:769] Sending connection context
0808 22:24:28.866010 (+ 100us) client_negotiation.cc:240] Negotiation successful
0808 22:24:28.866167 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":826,"thread_start_us":87,"threads_started":1}
I20260808 22:24:28.866420 32097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:28.855297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48524 (local address 127.9.176.193:34985)
0808 22:24:28.855681 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0808 22:24:28.855692 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 22:24:28.855740 (+ 48us) server_negotiation.cc:407] Connection header received
0808 22:24:28.855940 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:24:28.855945 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:24:28.856148 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:24:28.856273 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:24:28.857876 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:28.858859 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:28.860134 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:28.860454 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:28.860532 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:28.862765 (+ 2233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:24:28.862775 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:24:28.862787 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:24:28.862813 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:24:28.864548 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:24:28.865030 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:24:28.865036 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:24:28.865038 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:24:28.865133 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:24:28.865429 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:24:28.865432 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:24:28.865434 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:24:28.865703 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:24:28.865824 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0808 22:24:28.866126 (+ 302us) server_negotiation.cc:299] Negotiation successful
0808 22:24:28.866269 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":105,"threads_started":1}
I20260808 22:24:28.867347 31994 tablet_service.cc:1514] Processing DeleteTablet for tablet 67d030e843364bf2aacc35e81a36d472 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:24:28 UTC) from {username='kudu', principal='kudu/127.9.176.254@KRBTEST.COM'} at 127.0.0.1:48524
I20260808 22:24:28.868273 32098 tablet_replica.cc:330] stopping tablet replica
I20260808 22:24:28.868575 32098 raft_consensus.cc:2229] T 67d030e843364bf2aacc35e81a36d472 P c725f56e88ba4e649a6e854587c1da74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:24:28.868790 32098 raft_consensus.cc:2258] T 67d030e843364bf2aacc35e81a36d472 P c725f56e88ba4e649a6e854587c1da74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:24:28.869760 32098 ts_tablet_manager.cc:1905] T 67d030e843364bf2aacc35e81a36d472 P c725f56e88ba4e649a6e854587c1da74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:24:28.873219 32098 ts_tablet_manager.cc:1918] T 67d030e843364bf2aacc35e81a36d472 P c725f56e88ba4e649a6e854587c1da74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 22:24:28.873310 32098 log.cc:1196] T 67d030e843364bf2aacc35e81a36d472 P c725f56e88ba4e649a6e854587c1da74: Deleting WAL directory at /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-0/wal/wals/67d030e843364bf2aacc35e81a36d472
I20260808 22:24:28.873586 32098 ts_tablet_manager.cc:1939] T 67d030e843364bf2aacc35e81a36d472 P c725f56e88ba4e649a6e854587c1da74: Deleting consensus metadata
I20260808 22:24:28.875295 31805 catalog_manager.cc:4841] TS c725f56e88ba4e649a6e854587c1da74 (127.9.176.193:34985): tablet 67d030e843364bf2aacc35e81a36d472 (table test-table [id=db5d12dae627402a90f58548a4da4260]) successfully deleted
W20260808 22:24:28.954494 32091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:24:28.954752 32091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:24:28.954797 32091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:24:28.958673 32091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:24:28.958725 32091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:24:28.958782 32091 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.194
I20260808 22:24:28.963871 32091 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:33087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.194.keytab
--principal=kudu/127.9.176.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.194:37751
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.176.194
--webserver_port=44689
--tserver_master_addrs=127.9.176.254:38597
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32091
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.194
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:24:28.964826 32091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:24:28.965847 32091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 22:24:28.975797 32091 server_base.cc:1034] running on GCE node
W20260808 22:24:28.975764 32109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:24:28.975759 32105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:24:28.975759 32106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:24:28.976393 32091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:24:28.977056 32091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:24:28.978224 32091 hybrid_clock.cc:648] HybridClock initialized: now 1786227868978202 us; error 32 us; skew 500 ppm
Aug 08 22:24:28 dist-test-slave-ncp9 krb5kdc[30953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227868, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:24:28.980927 32091 init.cc:377] Logged in from keytab as kudu/127.9.176.194@KRBTEST.COM (short username kudu)
I20260808 22:24:28.981760 32091 webserver.cc:460] Webserver started at http://127.9.176.194:44689/ using document root <none> and password file <none>
I20260808 22:24:28.982296 32091 fs_manager.cc:359] Metadata directory not provided
I20260808 22:24:28.982383 32091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:24:28.986092 32091 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260808 22:24:28.988111 32119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:24:28.989367 32091 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 22:24:28.989470 32091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-1/wal
uuid: "b9bdb3f4a3d84fb2b48be7e89cd5672f"
format_stamp: "Formatted at 2026-08-08 22:24:27 on dist-test-slave-ncp9"
server_key: "d2e6403be0a1e99d03f9585ac7abcfdd"
server_key_iv: "2fe1edf73c12e43a6bc9fe61d223548d"
server_key_version: "encryptionkey@0"
I20260808 22:24:28.990020 32091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:24:29.010383 32091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:24:29.013252 32091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:24:29.013432 32091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:24:29.014024 32091 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:24:29.014861 32091 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:24:29.014909 32091 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:24:29.014961 32091 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:24:29.014990 32091 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:24:29.023341 32091 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.194:37751
I20260808 22:24:29.023379 32272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.194:37751 every 8 connection(s)
I20260808 22:24:29.024335 32091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
I20260808 22:24:29.031239 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 32091
I20260808 22:24:29.031700 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.195:44949
--local_ip_for_outbound_sockets=127.9.176.195
--tserver_master_addrs=127.9.176.254:38597
--webserver_port=39893
--webserver_interface=127.9.176.195
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.195.keytab
--principal=kudu/127.9.176.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:33087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 22:24:29 dist-test-slave-ncp9 krb5kdc[30953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227868, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.194@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
I20260808 22:24:29.037379 32089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:29.026104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.194:38003 (local address 127.9.176.254:38597)
0808 22:24:29.026273 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0808 22:24:29.026279 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 22:24:29.027291 (+ 1012us) server_negotiation.cc:407] Connection header received
0808 22:24:29.028204 (+ 913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:24:29.028208 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:24:29.028258 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:24:29.028335 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:24:29.029513 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:29.030043 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:29.031184 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:29.031528 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:29.031645 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:29.034069 (+ 2424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:24:29.034072 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:24:29.034074 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:24:29.034098 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:24:29.035415 (+ 1317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:24:29.035890 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:24:29.035893 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:24:29.035895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:24:29.035980 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:24:29.036296 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:24:29.036299 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:24:29.036301 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:24:29.036489 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:24:29.036574 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0808 22:24:29.037075 (+ 501us) server_negotiation.cc:299] Negotiation successful
0808 22:24:29.037211 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 22:24:29.038138 32279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:29.026479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:38597 (local address 127.9.176.194:38003)
0808 22:24:29.027142 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:29.027182 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 22:24:29.027981 (+ 799us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:24:29.028499 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:24:29.028506 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:24:29.028855 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:24:29.029385 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:24:29.029399 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:24:29.030140 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:24:29.030142 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:24:29.031050 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:24:29.031060 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:24:29.031687 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:24:29.031691 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:24:29.031870 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:29.032438 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:24:29.032454 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:24:29.033900 (+ 1446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:24:29.035532 (+ 1632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:24:29.035538 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:24:29.035552 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:24:29.035796 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:24:29.036062 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:24:29.036065 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:24:29.036067 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:24:29.036201 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:24:29.036590 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:24:29.036597 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:24:29.036823 (+ 226us) client_negotiation.cc:769] Sending connection context
0808 22:24:29.037022 (+ 199us) client_negotiation.cc:240] Negotiation successful
0808 22:24:29.037335 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":114,"threads_started":1}
I20260808 22:24:29.039402 32274 heartbeater.cc:347] Connected to a master server at 127.9.176.254:38597
I20260808 22:24:29.039626 32274 heartbeater.cc:464] Registering TS with master...
I20260808 22:24:29.040094 32274 heartbeater.cc:511] Master 127.9.176.254:38597 requested a full tablet report, sending...
I20260808 22:24:29.041182 31823 ts_manager.cc:194] Registered new tserver with Master: b9bdb3f4a3d84fb2b48be7e89cd5672f (127.9.176.194:37751)
I20260808 22:24:29.041960 31823 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.176.194@KRBTEST.COM'} at 127.9.176.194:38003
W20260808 22:24:29.144806 32280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:24:29.145112 32280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:24:29.145156 32280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:24:29.149099 32280 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:24:29.149154 32280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:24:29.149210 32280 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.195
I20260808 22:24:29.154645 32280 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:33087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.195.keytab
--principal=kudu/127.9.176.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.195:44949
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.176.195
--webserver_port=39893
--tserver_master_addrs=127.9.176.254:38597
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32280
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.195
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:24:29.155634 32280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:24:29.156596 32280 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:24:29.166010 32289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:24:29.166097 32291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:24:29.166075 32288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:24:29.166213 32280 server_base.cc:1034] running on GCE node
I20260808 22:24:29.166682 32280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:24:29.167284 32280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:24:29.168437 32280 hybrid_clock.cc:648] HybridClock initialized: now 1786227869168413 us; error 35 us; skew 500 ppm
Aug 08 22:24:29 dist-test-slave-ncp9 krb5kdc[30953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227869, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:24:29.171051 32280 init.cc:377] Logged in from keytab as kudu/127.9.176.195@KRBTEST.COM (short username kudu)
I20260808 22:24:29.172021 32280 webserver.cc:460] Webserver started at http://127.9.176.195:39893/ using document root <none> and password file <none>
I20260808 22:24:29.172530 32280 fs_manager.cc:359] Metadata directory not provided
I20260808 22:24:29.172609 32280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:24:29.176544 32280 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260808 22:24:29.178578 32301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:24:29.179723 32280 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 22:24:29.179829 32280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-2/wal
uuid: "e7519d2ff3a643fdbc922255349cc2d1"
format_stamp: "Formatted at 2026-08-08 22:24:28 on dist-test-slave-ncp9"
server_key: "ac780607a96c5bd1326617b7c3a4b343"
server_key_iv: "c95496a88a66d3ffaa5170a090833e43"
server_key_version: "encryptionkey@0"
I20260808 22:24:29.180344 32280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:24:29.191579 32280 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:24:29.194517 32280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:24:29.194681 32280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:24:29.195246 32280 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:24:29.196019 32280 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:24:29.196074 32280 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:24:29.196125 32280 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:24:29.196152 32280 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:24:29.204710 32280 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.195:44949
I20260808 22:24:29.204741 32472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.195:44949 every 8 connection(s)
I20260808 22:24:29.205674 32280 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
I20260808 22:24:29.208523 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 32280
Aug 08 22:24:29 dist-test-slave-ncp9 krb5kdc[30953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227869, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.195@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
I20260808 22:24:29.217284 32478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:29.209195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41076 (local address 127.9.176.254:38597)
0808 22:24:29.209476 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0808 22:24:29.209480 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:24:29.209561 (+ 81us) server_negotiation.cc:407] Connection header received
0808 22:24:29.209677 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:24:29.209679 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:24:29.209709 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:24:29.209777 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:24:29.210674 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:29.211279 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:29.212332 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:29.212566 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:29.212638 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:29.214270 (+ 1632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:24:29.214273 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:24:29.214275 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:24:29.214295 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:24:29.215690 (+ 1395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:24:29.216086 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:24:29.216089 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:24:29.216091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:24:29.216173 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:24:29.216410 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:24:29.216413 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:24:29.216415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:24:29.216590 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:24:29.216779 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0808 22:24:29.216974 (+ 195us) server_negotiation.cc:299] Negotiation successful
0808 22:24:29.217118 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":220,"thread_start_us":88,"threads_started":1}
I20260808 22:24:29.219074 32089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:29.207330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.195:36435 (local address 127.9.176.254:38597)
0808 22:24:29.207447 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0808 22:24:29.207452 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:24:29.208559 (+ 1107us) server_negotiation.cc:407] Connection header received
0808 22:24:29.209622 (+ 1063us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:24:29.209624 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:24:29.209674 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:24:29.209769 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:24:29.211127 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:29.211673 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:29.213152 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:29.213411 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:29.213485 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:29.215836 (+ 2351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:24:29.215839 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:24:29.215840 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:24:29.215861 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:24:29.217114 (+ 1253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:24:29.217685 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:24:29.217689 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:24:29.217691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:24:29.217780 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:24:29.218083 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:24:29.218087 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:24:29.218089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:24:29.218251 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:24:29.218334 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0808 22:24:29.218810 (+ 476us) server_negotiation.cc:299] Negotiation successful
0808 22:24:29.218922 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":39,"spinlock_wait_cycles":5888}
I20260808 22:24:29.219847 32475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:29.207657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:38597 (local address 127.9.176.195:36435)
0808 22:24:29.208382 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:29.208438 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0808 22:24:29.209406 (+ 968us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:24:29.209975 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:24:29.209985 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:24:29.210449 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:24:29.210994 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:24:29.211008 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:24:29.211793 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:24:29.211797 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:24:29.213042 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:24:29.213053 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:24:29.213562 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:24:29.213567 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:24:29.213778 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:29.214283 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:24:29.214299 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:24:29.215687 (+ 1388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:24:29.217246 (+ 1559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:24:29.217252 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:24:29.217266 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:24:29.217591 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:24:29.217856 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:24:29.217859 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:24:29.217861 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:24:29.217996 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:24:29.218352 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:24:29.218358 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:24:29.218589 (+ 231us) client_negotiation.cc:769] Sending connection context
0808 22:24:29.218747 (+ 158us) client_negotiation.cc:240] Negotiation successful
0808 22:24:29.219071 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":99,"threads_started":1}
I20260808 22:24:29.220930 32473 heartbeater.cc:347] Connected to a master server at 127.9.176.254:38597
I20260808 22:24:29.221194 32473 heartbeater.cc:464] Registering TS with master...
I20260808 22:24:29.221685 32473 heartbeater.cc:511] Master 127.9.176.254:38597 requested a full tablet report, sending...
I20260808 22:24:29.222694 31823 ts_manager.cc:194] Registered new tserver with Master: e7519d2ff3a643fdbc922255349cc2d1 (127.9.176.195:44949)
I20260808 22:24:29.223205 31823 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.176.195@KRBTEST.COM'} at 127.9.176.195:36435
I20260808 22:24:29.230432 9923 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 22:24:29.234380 32089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:29.231019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41078 (local address 127.9.176.254:38597)
0808 22:24:29.231152 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0808 22:24:29.231156 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:24:29.231319 (+ 163us) server_negotiation.cc:407] Connection header received
0808 22:24:29.231448 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:24:29.231450 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:24:29.231550 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:24:29.231631 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:24:29.231806 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:29.232288 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:29.233182 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:29.233383 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:29.233453 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:29.233684 (+ 231us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:24:29.234051 (+ 367us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:24:29.234132 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0808 22:24:29.234213 (+ 81us) server_negotiation.cc:299] Negotiation successful
0808 22:24:29.234257 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 22:24:29.234979 31823 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:41078:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 22:24:29.238154 31823 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 22:24:29.245724 31994 tablet_service.cc:1467] Processing CreateTablet for tablet c358d840f37b4fc1b57911428d8fb659 (DEFAULT_TABLE table=test-table [id=c3f0b788ca29447e8dd1a545341293fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:24:29.246425 31994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c358d840f37b4fc1b57911428d8fb659. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:24:29.249482 31928 tablet_bootstrap.cc:492] T c358d840f37b4fc1b57911428d8fb659 P c725f56e88ba4e649a6e854587c1da74: Bootstrap starting.
I20260808 22:24:29.250075 32096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:29.244552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.195:44949 (local address 127.0.0.1:41804)
0808 22:24:29.244670 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:29.244687 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 22:24:29.244775 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:24:29.245746 (+ 971us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:24:29.245749 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:24:29.245766 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:24:29.245866 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:24:29.245872 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:24:29.247380 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:24:29.247383 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:24:29.248247 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:24:29.248259 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:24:29.249622 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:24:29.249627 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:24:29.249774 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:29.249826 (+ 52us) client_negotiation.cc:769] Sending connection context
0808 22:24:29.249891 (+ 65us) client_negotiation.cc:240] Negotiation successful
0808 22:24:29.249973 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260808 22:24:29.250370 31928 tablet_bootstrap.cc:654] T c358d840f37b4fc1b57911428d8fb659 P c725f56e88ba4e649a6e854587c1da74: Neither blocks nor log segments found. Creating new log.
I20260808 22:24:29.250953 32484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:29.244743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41804 (local address 127.9.176.195:44949)
0808 22:24:29.245105 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0808 22:24:29.245122 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 22:24:29.245149 (+ 27us) server_negotiation.cc:407] Connection header received
0808 22:24:29.245235 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:24:29.245241 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:24:29.245524 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:24:29.245752 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:24:29.245968 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:29.247217 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:29.248359 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:29.249471 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:29.250356 (+ 885us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:29.250507 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0808 22:24:29.250558 (+ 51us) server_negotiation.cc:299] Negotiation successful
0808 22:24:29.250805 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":144,"threads_started":1}
I20260808 22:24:29.250988 32485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:29.244838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.194:37751 (local address 127.0.0.1:56694)
0808 22:24:29.245178 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:29.245191 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 22:24:29.245283 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:24:29.246237 (+ 954us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:24:29.246240 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:24:29.246258 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:24:29.246344 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:24:29.246350 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:24:29.248376 (+ 2026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:24:29.248379 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:24:29.249225 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:24:29.249236 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:24:29.250436 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:24:29.250439 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:24:29.250523 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:29.250549 (+ 26us) client_negotiation.cc:769] Sending connection context
0808 22:24:29.250622 (+ 73us) client_negotiation.cc:240] Negotiation successful
0808 22:24:29.250689 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":138,"threads_started":1}
I20260808 22:24:29.251313 32486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:29.245319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56694 (local address 127.9.176.194:37751)
0808 22:24:29.245783 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0808 22:24:29.245795 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 22:24:29.245817 (+ 22us) server_negotiation.cc:407] Connection header received
0808 22:24:29.245888 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:24:29.245893 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:24:29.246102 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:24:29.246252 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:24:29.246436 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:29.247248 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:29.249344 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:29.250118 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:29.250219 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:29.250365 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0808 22:24:29.250693 (+ 328us) server_negotiation.cc:299] Negotiation successful
0808 22:24:29.250793 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":159,"threads_started":1}
I20260808 22:24:29.252446 31928 tablet_bootstrap.cc:492] T c358d840f37b4fc1b57911428d8fb659 P c725f56e88ba4e649a6e854587c1da74: No bootstrap required, opened a new log
I20260808 22:24:29.252521 31928 ts_tablet_manager.cc:1397] T c358d840f37b4fc1b57911428d8fb659 P c725f56e88ba4e649a6e854587c1da74: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260808 22:24:29.252939 31928 raft_consensus.cc:348] T c358d840f37b4fc1b57911428d8fb659 P c725f56e88ba4e649a6e854587c1da74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7519d2ff3a643fdbc922255349cc2d1" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 44949 } } peers { permanent_uuid: "b9bdb3f4a3d84fb2b48be7e89cd5672f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 37751 } } peers { permanent_uuid: "c725f56e88ba4e649a6e854587c1da74" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34985 } }
I20260808 22:24:29.253041 31928 raft_consensus.cc:374] T c358d840f37b4fc1b57911428d8fb659 P c725f56e88ba4e649a6e854587c1da74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:24:29.253078 31928 raft_consensus.cc:729] T c358d840f37b4fc1b57911428d8fb659 P c725f56e88ba4e649a6e854587c1da74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c725f56e88ba4e649a6e854587c1da74, State: Initialized, Role: FOLLOWER
I20260808 22:24:29.253207 31928 consensus_queue.cc:260] T c358d840f37b4fc1b57911428d8fb659 P c725f56e88ba4e649a6e854587c1da74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7519d2ff3a643fdbc922255349cc2d1" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 44949 } } peers { permanent_uuid: "b9bdb3f4a3d84fb2b48be7e89cd5672f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 37751 } } peers { permanent_uuid: "c725f56e88ba4e649a6e854587c1da74" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34985 } }
I20260808 22:24:29.253943 31928 ts_tablet_manager.cc:1428] T c358d840f37b4fc1b57911428d8fb659 P c725f56e88ba4e649a6e854587c1da74: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 22:24:29.254356 32380 tablet_service.cc:1467] Processing CreateTablet for tablet c358d840f37b4fc1b57911428d8fb659 (DEFAULT_TABLE table=test-table [id=c3f0b788ca29447e8dd1a545341293fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:24:29.254983 32189 tablet_service.cc:1467] Processing CreateTablet for tablet c358d840f37b4fc1b57911428d8fb659 (DEFAULT_TABLE table=test-table [id=c3f0b788ca29447e8dd1a545341293fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:24:29.255522 32380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c358d840f37b4fc1b57911428d8fb659. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:24:29.256039 32189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c358d840f37b4fc1b57911428d8fb659. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:24:29.263999 32490 tablet_bootstrap.cc:492] T c358d840f37b4fc1b57911428d8fb659 P e7519d2ff3a643fdbc922255349cc2d1: Bootstrap starting.
I20260808 22:24:29.263999 32489 tablet_bootstrap.cc:492] T c358d840f37b4fc1b57911428d8fb659 P b9bdb3f4a3d84fb2b48be7e89cd5672f: Bootstrap starting.
I20260808 22:24:29.266789 32490 tablet_bootstrap.cc:654] T c358d840f37b4fc1b57911428d8fb659 P e7519d2ff3a643fdbc922255349cc2d1: Neither blocks nor log segments found. Creating new log.
I20260808 22:24:29.266789 32489 tablet_bootstrap.cc:654] T c358d840f37b4fc1b57911428d8fb659 P b9bdb3f4a3d84fb2b48be7e89cd5672f: Neither blocks nor log segments found. Creating new log.
I20260808 22:24:29.267699 32489 log.cc:824] T c358d840f37b4fc1b57911428d8fb659 P b9bdb3f4a3d84fb2b48be7e89cd5672f: Log is configured to *not* fsync() on all Append() calls
I20260808 22:24:29.267699 32490 log.cc:824] T c358d840f37b4fc1b57911428d8fb659 P e7519d2ff3a643fdbc922255349cc2d1: Log is configured to *not* fsync() on all Append() calls
I20260808 22:24:29.269948 32489 tablet_bootstrap.cc:492] T c358d840f37b4fc1b57911428d8fb659 P b9bdb3f4a3d84fb2b48be7e89cd5672f: No bootstrap required, opened a new log
I20260808 22:24:29.269948 32490 tablet_bootstrap.cc:492] T c358d840f37b4fc1b57911428d8fb659 P e7519d2ff3a643fdbc922255349cc2d1: No bootstrap required, opened a new log
I20260808 22:24:29.270150 32489 ts_tablet_manager.cc:1397] T c358d840f37b4fc1b57911428d8fb659 P b9bdb3f4a3d84fb2b48be7e89cd5672f: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260808 22:24:29.270150 32490 ts_tablet_manager.cc:1397] T c358d840f37b4fc1b57911428d8fb659 P e7519d2ff3a643fdbc922255349cc2d1: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 22:24:29.273988 32489 raft_consensus.cc:348] T c358d840f37b4fc1b57911428d8fb659 P b9bdb3f4a3d84fb2b48be7e89cd5672f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7519d2ff3a643fdbc922255349cc2d1" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 44949 } } peers { permanent_uuid: "b9bdb3f4a3d84fb2b48be7e89cd5672f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 37751 } } peers { permanent_uuid: "c725f56e88ba4e649a6e854587c1da74" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34985 } }
I20260808 22:24:29.273988 32490 raft_consensus.cc:348] T c358d840f37b4fc1b57911428d8fb659 P e7519d2ff3a643fdbc922255349cc2d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7519d2ff3a643fdbc922255349cc2d1" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 44949 } } peers { permanent_uuid: "b9bdb3f4a3d84fb2b48be7e89cd5672f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 37751 } } peers { permanent_uuid: "c725f56e88ba4e649a6e854587c1da74" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34985 } }
I20260808 22:24:29.274276 32490 raft_consensus.cc:374] T c358d840f37b4fc1b57911428d8fb659 P e7519d2ff3a643fdbc922255349cc2d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:24:29.274276 32489 raft_consensus.cc:374] T c358d840f37b4fc1b57911428d8fb659 P b9bdb3f4a3d84fb2b48be7e89cd5672f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:24:29.274339 32489 raft_consensus.cc:729] T c358d840f37b4fc1b57911428d8fb659 P b9bdb3f4a3d84fb2b48be7e89cd5672f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9bdb3f4a3d84fb2b48be7e89cd5672f, State: Initialized, Role: FOLLOWER
I20260808 22:24:29.274346 32490 raft_consensus.cc:729] T c358d840f37b4fc1b57911428d8fb659 P e7519d2ff3a643fdbc922255349cc2d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7519d2ff3a643fdbc922255349cc2d1, State: Initialized, Role: FOLLOWER
I20260808 22:24:29.274807 32489 consensus_queue.cc:260] T c358d840f37b4fc1b57911428d8fb659 P b9bdb3f4a3d84fb2b48be7e89cd5672f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7519d2ff3a643fdbc922255349cc2d1" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 44949 } } peers { permanent_uuid: "b9bdb3f4a3d84fb2b48be7e89cd5672f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 37751 } } peers { permanent_uuid: "c725f56e88ba4e649a6e854587c1da74" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34985 } }
I20260808 22:24:29.274870 32490 consensus_queue.cc:260] T c358d840f37b4fc1b57911428d8fb659 P e7519d2ff3a643fdbc922255349cc2d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7519d2ff3a643fdbc922255349cc2d1" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 44949 } } peers { permanent_uuid: "b9bdb3f4a3d84fb2b48be7e89cd5672f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 37751 } } peers { permanent_uuid: "c725f56e88ba4e649a6e854587c1da74" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34985 } }
I20260808 22:24:29.275655 32473 heartbeater.cc:503] Master 127.9.176.254:38597 was elected leader, sending a full tablet report...
I20260808 22:24:29.275825 32490 ts_tablet_manager.cc:1428] T c358d840f37b4fc1b57911428d8fb659 P e7519d2ff3a643fdbc922255349cc2d1: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260808 22:24:29.275825 32489 ts_tablet_manager.cc:1428] T c358d840f37b4fc1b57911428d8fb659 P b9bdb3f4a3d84fb2b48be7e89cd5672f: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260808 22:24:29.276160 32274 heartbeater.cc:503] Master 127.9.176.254:38597 was elected leader, sending a full tablet report...
W20260808 22:24:29.330495 32088 tablet.cc:2369] T c358d840f37b4fc1b57911428d8fb659 P c725f56e88ba4e649a6e854587c1da74: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 22:24:29.456641 32474 tablet.cc:2369] T c358d840f37b4fc1b57911428d8fb659 P e7519d2ff3a643fdbc922255349cc2d1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 22:24:29.525485 32276 tablet.cc:2369] T c358d840f37b4fc1b57911428d8fb659 P b9bdb3f4a3d84fb2b48be7e89cd5672f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 22:24:29.552781 32496 raft_consensus.cc:482] T c358d840f37b4fc1b57911428d8fb659 P e7519d2ff3a643fdbc922255349cc2d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 22:24:29.553074 32496 raft_consensus.cc:504] T c358d840f37b4fc1b57911428d8fb659 P e7519d2ff3a643fdbc922255349cc2d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7519d2ff3a643fdbc922255349cc2d1" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 44949 } } peers { permanent_uuid: "b9bdb3f4a3d84fb2b48be7e89cd5672f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 37751 } } peers { permanent_uuid: "c725f56e88ba4e649a6e854587c1da74" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34985 } }
I20260808 22:24:29.554296 32496 leader_election.cc:288] T c358d840f37b4fc1b57911428d8fb659 P e7519d2ff3a643fdbc922255349cc2d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9bdb3f4a3d84fb2b48be7e89cd5672f (127.9.176.194:37751), c725f56e88ba4e649a6e854587c1da74 (127.9.176.193:34985)
I20260808 22:24:29.557806 32475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:29.554557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.194:37751 (local address 127.9.176.195:35593)
0808 22:24:29.554751 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:29.554765 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 22:24:29.554887 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:24:29.555116 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:24:29.555119 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:24:29.555146 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:24:29.555250 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:24:29.555255 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:24:29.555975 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:24:29.555978 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:24:29.556746 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:24:29.556755 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:24:29.557408 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:24:29.557412 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:24:29.557517 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:29.557536 (+ 19us) client_negotiation.cc:769] Sending connection context
0808 22:24:29.557602 (+ 66us) client_negotiation.cc:240] Negotiation successful
0808 22:24:29.557674 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"spinlock_wait_cycles":768}
I20260808 22:24:29.557806 32486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:29.554642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.195:35593 (local address 127.9.176.194:37751)
0808 22:24:29.554776 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0808 22:24:29.554782 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 22:24:29.554886 (+ 104us) server_negotiation.cc:407] Connection header received
0808 22:24:29.554980 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:24:29.554983 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:24:29.555029 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:24:29.555091 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:24:29.555361 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:29.555867 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:29.556875 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:29.557297 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:29.557407 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:29.557462 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0808 22:24:29.557629 (+ 167us) server_negotiation.cc:299] Negotiation successful
0808 22:24:29.557696 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 22:24:29.558315 32498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:29.554721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.195:55013 (local address 127.9.176.193:34985)
0808 22:24:29.555033 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0808 22:24:29.555040 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 22:24:29.555104 (+ 64us) server_negotiation.cc:407] Connection header received
0808 22:24:29.555240 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:24:29.555244 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:24:29.555303 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:24:29.555405 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:24:29.555687 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:29.556191 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:29.557308 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:29.557751 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:29.557874 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:29.558040 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0808 22:24:29.558121 (+ 81us) server_negotiation.cc:299] Negotiation successful
0808 22:24:29.558193 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":74,"threads_started":1}
I20260808 22:24:29.558318 32497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:29.554609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.193:34985 (local address 127.9.176.195:55013)
0808 22:24:29.554999 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:29.555014 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 22:24:29.555137 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:24:29.555426 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:24:29.555429 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:24:29.555446 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:24:29.555575 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:24:29.555582 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:24:29.556306 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:24:29.556310 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:24:29.557179 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:24:29.557190 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:24:29.557874 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:24:29.557878 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:24:29.557986 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:29.558011 (+ 25us) client_negotiation.cc:769] Sending connection context
0808 22:24:29.558099 (+ 88us) client_negotiation.cc:240] Negotiation successful
0808 22:24:29.558168 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":124,"threads_started":1}
I20260808 22:24:29.558467 32215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c358d840f37b4fc1b57911428d8fb659" candidate_uuid: "e7519d2ff3a643fdbc922255349cc2d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9bdb3f4a3d84fb2b48be7e89cd5672f" is_pre_election: true
I20260808 22:24:29.558808 32215 raft_consensus.cc:2454] T c358d840f37b4fc1b57911428d8fb659 P b9bdb3f4a3d84fb2b48be7e89cd5672f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7519d2ff3a643fdbc922255349cc2d1 in term 0.
I20260808 22:24:29.558890 32022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c358d840f37b4fc1b57911428d8fb659" candidate_uuid: "e7519d2ff3a643fdbc922255349cc2d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c725f56e88ba4e649a6e854587c1da74" is_pre_election: true
I20260808 22:24:29.559203 32022 raft_consensus.cc:2454] T c358d840f37b4fc1b57911428d8fb659 P c725f56e88ba4e649a6e854587c1da74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7519d2ff3a643fdbc922255349cc2d1 in term 0.
I20260808 22:24:29.559319 32302 leader_election.cc:302] T c358d840f37b4fc1b57911428d8fb659 P e7519d2ff3a643fdbc922255349cc2d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9bdb3f4a3d84fb2b48be7e89cd5672f, e7519d2ff3a643fdbc922255349cc2d1; no voters:
I20260808 22:24:29.559593 32496 raft_consensus.cc:2781] T c358d840f37b4fc1b57911428d8fb659 P e7519d2ff3a643fdbc922255349cc2d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 22:24:29.559672 32496 raft_consensus.cc:482] T c358d840f37b4fc1b57911428d8fb659 P e7519d2ff3a643fdbc922255349cc2d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 22:24:29.559716 32496 raft_consensus.cc:3037] T c358d840f37b4fc1b57911428d8fb659 P e7519d2ff3a643fdbc922255349cc2d1 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:24:29.560734 32496 raft_consensus.cc:504] T c358d840f37b4fc1b57911428d8fb659 P e7519d2ff3a643fdbc922255349cc2d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7519d2ff3a643fdbc922255349cc2d1" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 44949 } } peers { permanent_uuid: "b9bdb3f4a3d84fb2b48be7e89cd5672f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 37751 } } peers { permanent_uuid: "c725f56e88ba4e649a6e854587c1da74" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34985 } }
I20260808 22:24:29.561085 32496 leader_election.cc:288] T c358d840f37b4fc1b57911428d8fb659 P e7519d2ff3a643fdbc922255349cc2d1 [CANDIDATE]: Term 1 election: Requested vote from peers b9bdb3f4a3d84fb2b48be7e89cd5672f (127.9.176.194:37751), c725f56e88ba4e649a6e854587c1da74 (127.9.176.193:34985)
I20260808 22:24:29.561443 32022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c358d840f37b4fc1b57911428d8fb659" candidate_uuid: "e7519d2ff3a643fdbc922255349cc2d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c725f56e88ba4e649a6e854587c1da74"
I20260808 22:24:29.561468 32215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c358d840f37b4fc1b57911428d8fb659" candidate_uuid: "e7519d2ff3a643fdbc922255349cc2d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9bdb3f4a3d84fb2b48be7e89cd5672f"
I20260808 22:24:29.561586 32022 raft_consensus.cc:3037] T c358d840f37b4fc1b57911428d8fb659 P c725f56e88ba4e649a6e854587c1da74 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:24:29.561586 32215 raft_consensus.cc:3037] T c358d840f37b4fc1b57911428d8fb659 P b9bdb3f4a3d84fb2b48be7e89cd5672f [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:24:29.562919 32022 raft_consensus.cc:2454] T c358d840f37b4fc1b57911428d8fb659 P c725f56e88ba4e649a6e854587c1da74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7519d2ff3a643fdbc922255349cc2d1 in term 1.
I20260808 22:24:29.562919 32215 raft_consensus.cc:2454] T c358d840f37b4fc1b57911428d8fb659 P b9bdb3f4a3d84fb2b48be7e89cd5672f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7519d2ff3a643fdbc922255349cc2d1 in term 1.
I20260808 22:24:29.563196 32302 leader_election.cc:302] T c358d840f37b4fc1b57911428d8fb659 P e7519d2ff3a643fdbc922255349cc2d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9bdb3f4a3d84fb2b48be7e89cd5672f, e7519d2ff3a643fdbc922255349cc2d1; no voters:
I20260808 22:24:29.563366 32496 raft_consensus.cc:2781] T c358d840f37b4fc1b57911428d8fb659 P e7519d2ff3a643fdbc922255349cc2d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:24:29.563572 32496 raft_consensus.cc:686] T c358d840f37b4fc1b57911428d8fb659 P e7519d2ff3a643fdbc922255349cc2d1 [term 1 LEADER]: Becoming Leader. State: Replica: e7519d2ff3a643fdbc922255349cc2d1, State: Running, Role: LEADER
I20260808 22:24:29.563889 32496 consensus_queue.cc:237] T c358d840f37b4fc1b57911428d8fb659 P e7519d2ff3a643fdbc922255349cc2d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7519d2ff3a643fdbc922255349cc2d1" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 44949 } } peers { permanent_uuid: "b9bdb3f4a3d84fb2b48be7e89cd5672f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 37751 } } peers { permanent_uuid: "c725f56e88ba4e649a6e854587c1da74" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34985 } }
I20260808 22:24:29.566982 31823 catalog_manager.cc:5495] T c358d840f37b4fc1b57911428d8fb659 P e7519d2ff3a643fdbc922255349cc2d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to e7519d2ff3a643fdbc922255349cc2d1 (127.9.176.195). New cstate: current_term: 1 leader_uuid: "e7519d2ff3a643fdbc922255349cc2d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7519d2ff3a643fdbc922255349cc2d1" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 44949 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9bdb3f4a3d84fb2b48be7e89cd5672f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 37751 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c725f56e88ba4e649a6e854587c1da74" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34985 } health_report { overall_health: UNKNOWN } } }
I20260808 22:24:29.611663 32484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:29.608442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41810 (local address 127.9.176.195:44949)
0808 22:24:29.608572 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0808 22:24:29.608576 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:24:29.608590 (+ 14us) server_negotiation.cc:407] Connection header received
0808 22:24:29.608703 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:24:29.608706 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:24:29.608760 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:24:29.608847 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:24:29.609093 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:29.609606 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:29.610366 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:29.610570 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:29.610636 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:29.610943 (+ 307us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:24:29.611347 (+ 404us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:24:29.611430 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0808 22:24:29.611494 (+ 64us) server_negotiation.cc:299] Negotiation successful
0808 22:24:29.611555 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 22:24:29.617328 32022 raft_consensus.cc:1261] T c358d840f37b4fc1b57911428d8fb659 P c725f56e88ba4e649a6e854587c1da74 [term 1 FOLLOWER]: Refusing update from remote peer e7519d2ff3a643fdbc922255349cc2d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:24:29.617333 32215 raft_consensus.cc:1261] T c358d840f37b4fc1b57911428d8fb659 P b9bdb3f4a3d84fb2b48be7e89cd5672f [term 1 FOLLOWER]: Refusing update from remote peer e7519d2ff3a643fdbc922255349cc2d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:24:29.618000 32496 consensus_queue.cc:1035] T c358d840f37b4fc1b57911428d8fb659 P e7519d2ff3a643fdbc922255349cc2d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9bdb3f4a3d84fb2b48be7e89cd5672f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 37751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:24:29.618185 32501 consensus_queue.cc:1035] T c358d840f37b4fc1b57911428d8fb659 P e7519d2ff3a643fdbc922255349cc2d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c725f56e88ba4e649a6e854587c1da74" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:24:29.623885 32507 mvcc.cc:205] Tried to move back new op lower bound from 7316389353945538560 to 7316389353735888896. Current Snapshot: MvccSnapshot[applied={T|T < 7316389353945538560}]
I20260808 22:24:29.628969 32508 mvcc.cc:205] Tried to move back new op lower bound from 7316389353945538560 to 7316389353735888896. Current Snapshot: MvccSnapshot[applied={T|T < 7316389353945538560 or (T in {7316389353945538560})}]
I20260808 22:24:29.631687 31824 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:41078:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:24:29.631842 31824 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:41078:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:24:29.633965 31824 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 35e08196664442e9b7f686512baa70f5: Sending DeleteTablet for 3 replicas of tablet c358d840f37b4fc1b57911428d8fb659
I20260808 22:24:29.634593 32380 tablet_service.cc:1514] Processing DeleteTablet for tablet c358d840f37b4fc1b57911428d8fb659 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:24:29 UTC) from {username='kudu', principal='kudu/127.9.176.254@KRBTEST.COM'} at 127.0.0.1:41804
I20260808 22:24:29.634763 32189 tablet_service.cc:1514] Processing DeleteTablet for tablet c358d840f37b4fc1b57911428d8fb659 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:24:29 UTC) from {username='kudu', principal='kudu/127.9.176.254@KRBTEST.COM'} at 127.0.0.1:56694
I20260808 22:24:29.635216 32519 tablet_replica.cc:330] T c358d840f37b4fc1b57911428d8fb659 P b9bdb3f4a3d84fb2b48be7e89cd5672f: stopping tablet replica
I20260808 22:24:29.635424 31994 tablet_service.cc:1514] Processing DeleteTablet for tablet c358d840f37b4fc1b57911428d8fb659 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:24:29 UTC) from {username='kudu', principal='kudu/127.9.176.254@KRBTEST.COM'} at 127.0.0.1:48524
I20260808 22:24:29.635650 32519 raft_consensus.cc:2229] T c358d840f37b4fc1b57911428d8fb659 P b9bdb3f4a3d84fb2b48be7e89cd5672f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:24:29.635957 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 31874
I20260808 22:24:29.636070 32519 raft_consensus.cc:2258] T c358d840f37b4fc1b57911428d8fb659 P b9bdb3f4a3d84fb2b48be7e89cd5672f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:24:29.648609 32519 ts_tablet_manager.cc:1905] T c358d840f37b4fc1b57911428d8fb659 P b9bdb3f4a3d84fb2b48be7e89cd5672f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:24:29.651247 32519 ts_tablet_manager.cc:1918] T c358d840f37b4fc1b57911428d8fb659 P b9bdb3f4a3d84fb2b48be7e89cd5672f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 22:24:29.651352 32519 log.cc:1196] T c358d840f37b4fc1b57911428d8fb659 P b9bdb3f4a3d84fb2b48be7e89cd5672f: Deleting WAL directory at /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-1/wal/wals/c358d840f37b4fc1b57911428d8fb659
I20260808 22:24:29.651662 32519 ts_tablet_manager.cc:1939] T c358d840f37b4fc1b57911428d8fb659 P b9bdb3f4a3d84fb2b48be7e89cd5672f: Deleting consensus metadata
I20260808 22:24:29.652868 31803 catalog_manager.cc:4841] TS b9bdb3f4a3d84fb2b48be7e89cd5672f (127.9.176.194:37751): tablet c358d840f37b4fc1b57911428d8fb659 (table test-table [id=c3f0b788ca29447e8dd1a545341293fe]) successfully deleted
W20260808 22:24:29.657184 31805 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.176.193:34985 (error 108)
I20260808 22:24:29.657536 32516 tablet_replica.cc:330] T c358d840f37b4fc1b57911428d8fb659 P e7519d2ff3a643fdbc922255349cc2d1: stopping tablet replica
I20260808 22:24:29.657933 32516 raft_consensus.cc:2229] T c358d840f37b4fc1b57911428d8fb659 P e7519d2ff3a643fdbc922255349cc2d1 [term 1 LEADER]: Raft consensus shutting down.
I20260808 22:24:29.658355 32485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:29.658080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.193:34985 (local address 127.0.0.1:48534)
0808 22:24:29.658193 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:29.658266 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.193:34985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260808 22:24:29.658346 32516 raft_consensus.cc:2258] T c358d840f37b4fc1b57911428d8fb659 P e7519d2ff3a643fdbc922255349cc2d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:24:29.658485 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 32091
W20260808 22:24:29.658655 31805 catalog_manager.cc:4584] TS c725f56e88ba4e649a6e854587c1da74 (127.9.176.193:34985): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c358d840f37b4fc1b57911428d8fb659: Network error: Client connection negotiation failed: client connection to 127.9.176.193:34985: connect: Connection refused (error 111)
I20260808 22:24:29.660177 32516 ts_tablet_manager.cc:1905] T c358d840f37b4fc1b57911428d8fb659 P e7519d2ff3a643fdbc922255349cc2d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:24:29.663378 32516 ts_tablet_manager.cc:1918] T c358d840f37b4fc1b57911428d8fb659 P e7519d2ff3a643fdbc922255349cc2d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 22:24:29.663568 32516 log.cc:1196] T c358d840f37b4fc1b57911428d8fb659 P e7519d2ff3a643fdbc922255349cc2d1: Deleting WAL directory at /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-2/wal/wals/c358d840f37b4fc1b57911428d8fb659
I20260808 22:24:29.664099 32516 ts_tablet_manager.cc:1939] T c358d840f37b4fc1b57911428d8fb659 P e7519d2ff3a643fdbc922255349cc2d1: Deleting consensus metadata
I20260808 22:24:29.665746 31806 catalog_manager.cc:4841] TS e7519d2ff3a643fdbc922255349cc2d1 (127.9.176.195:44949): tablet c358d840f37b4fc1b57911428d8fb659 (table test-table [id=c3f0b788ca29447e8dd1a545341293fe]) successfully deleted
I20260808 22:24:29.680037 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 32280
I20260808 22:24:29.689880 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 31770
I20260808 22:24:29.705451 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:38597
--webserver_interface=127.9.176.254
--webserver_port=40381
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab
--principal=kudu/127.9.176.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:33087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.176.254:38597
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:24:29.816491 32523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:24:29.816746 32523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:24:29.816810 32523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:24:29.820693 32523 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:24:29.820752 32523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:24:29.820789 32523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:24:29.820825 32523 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:24:29.820859 32523 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:24:29.825987 32523 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:33087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.176.254:38597
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab
--principal=kudu/127.9.176.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:38597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.176.254
--webserver_port=40381
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32523
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:24:29.826942 32523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:24:29.827895 32523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:24:29.836110 32535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:24:29.836229 32523 server_base.cc:1034] running on GCE node
W20260808 22:24:29.836165 32533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:24:29.836169 32532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:24:29.836879 32523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:24:29.837913 32523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:24:29.839071 32523 hybrid_clock.cc:648] HybridClock initialized: now 1786227869839054 us; error 29 us; skew 500 ppm
Aug 08 22:24:29 dist-test-slave-ncp9 krb5kdc[30953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227869, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:24:29.841771 32523 init.cc:377] Logged in from keytab as kudu/127.9.176.254@KRBTEST.COM (short username kudu)
I20260808 22:24:29.842593 32523 webserver.cc:460] Webserver started at http://127.9.176.254:40381/ using document root <none> and password file <none>
I20260808 22:24:29.843089 32523 fs_manager.cc:359] Metadata directory not provided
I20260808 22:24:29.843154 32523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:24:29.846830 32523 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260808 22:24:29.848810 32544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:24:29.849980 32523 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 22:24:29.850097 32523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/master-0/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/master-0/wal
uuid: "35e08196664442e9b7f686512baa70f5"
format_stamp: "Formatted at 2026-08-08 22:24:27 on dist-test-slave-ncp9"
server_key: "7653e1e631037b8f7bd49e7fad710b74"
server_key_iv: "86cf734e2347e21a839717c26f3d69a7"
server_key_version: "encryptionkey@0"
I20260808 22:24:29.850615 32523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:24:29.868065 32523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:24:29.871153 32523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:24:29.871346 32523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:24:29.878182 32523 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.254:38597
I20260808 22:24:29.878489 32624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.254:38597 every 8 connection(s)
I20260808 22:24:29.879220 32523 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
I20260808 22:24:29.881664 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 32523
I20260808 22:24:29.882071 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.193:34985
--local_ip_for_outbound_sockets=127.9.176.193
--tserver_master_addrs=127.9.176.254:38597
--webserver_port=42243
--webserver_interface=127.9.176.193
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.193.keytab
--principal=kudu/127.9.176.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:33087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 22:24:29.883631 32625 sys_catalog.cc:263] Verifying existing consensus state
I20260808 22:24:29.885795 32625 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35e08196664442e9b7f686512baa70f5: Bootstrap starting.
I20260808 22:24:29.906917 32625 log.cc:824] T 00000000000000000000000000000000 P 35e08196664442e9b7f686512baa70f5: Log is configured to *not* fsync() on all Append() calls
I20260808 22:24:29.917887 32625 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35e08196664442e9b7f686512baa70f5: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 22:24:29.918229 32625 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35e08196664442e9b7f686512baa70f5: Bootstrap complete.
I20260808 22:24:29.922068 32625 raft_consensus.cc:348] T 00000000000000000000000000000000 P 35e08196664442e9b7f686512baa70f5 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35e08196664442e9b7f686512baa70f5" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 38597 } }
I20260808 22:24:29.922466 32625 raft_consensus.cc:729] T 00000000000000000000000000000000 P 35e08196664442e9b7f686512baa70f5 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35e08196664442e9b7f686512baa70f5, State: Initialized, Role: FOLLOWER
I20260808 22:24:29.922902 32625 consensus_queue.cc:260] T 00000000000000000000000000000000 P 35e08196664442e9b7f686512baa70f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35e08196664442e9b7f686512baa70f5" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 38597 } }
I20260808 22:24:29.923022 32625 raft_consensus.cc:388] T 00000000000000000000000000000000 P 35e08196664442e9b7f686512baa70f5 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:24:29.923094 32625 raft_consensus.cc:482] T 00000000000000000000000000000000 P 35e08196664442e9b7f686512baa70f5 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:24:29.923187 32625 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 35e08196664442e9b7f686512baa70f5 [term 2 FOLLOWER]: Advancing to term 3
I20260808 22:24:29.924968 32625 raft_consensus.cc:504] T 00000000000000000000000000000000 P 35e08196664442e9b7f686512baa70f5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35e08196664442e9b7f686512baa70f5" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 38597 } }
I20260808 22:24:29.925282 32625 leader_election.cc:302] T 00000000000000000000000000000000 P 35e08196664442e9b7f686512baa70f5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 35e08196664442e9b7f686512baa70f5; no voters:
I20260808 22:24:29.925586 32625 leader_election.cc:288] T 00000000000000000000000000000000 P 35e08196664442e9b7f686512baa70f5 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260808 22:24:29.925683 32630 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 35e08196664442e9b7f686512baa70f5 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 22:24:29.925942 32630 raft_consensus.cc:686] T 00000000000000000000000000000000 P 35e08196664442e9b7f686512baa70f5 [term 3 LEADER]: Becoming Leader. State: Replica: 35e08196664442e9b7f686512baa70f5, State: Running, Role: LEADER
I20260808 22:24:29.926236 32630 consensus_queue.cc:237] T 00000000000000000000000000000000 P 35e08196664442e9b7f686512baa70f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35e08196664442e9b7f686512baa70f5" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 38597 } }
I20260808 22:24:29.926882 32625 sys_catalog.cc:564] T 00000000000000000000000000000000 P 35e08196664442e9b7f686512baa70f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:24:29.927547 32632 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35e08196664442e9b7f686512baa70f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "35e08196664442e9b7f686512baa70f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35e08196664442e9b7f686512baa70f5" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 38597 } } }
I20260808 22:24:29.927696 32632 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35e08196664442e9b7f686512baa70f5 [sys.catalog]: This master's current role is: LEADER
I20260808 22:24:29.928249 32633 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35e08196664442e9b7f686512baa70f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 35e08196664442e9b7f686512baa70f5. Latest consensus state: current_term: 3 leader_uuid: "35e08196664442e9b7f686512baa70f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35e08196664442e9b7f686512baa70f5" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 38597 } } }
I20260808 22:24:29.928390 32633 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35e08196664442e9b7f686512baa70f5 [sys.catalog]: This master's current role is: LEADER
I20260808 22:24:29.928750 32642 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:24:29.933676 32642 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:24:29.934319 32642 catalog_manager.cc:1259] Loaded cluster ID: 9f79495c212b4ff4a668128b2376b2af
I20260808 22:24:29.934379 32642 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260808 22:24:29.935544 32642 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f6f1657b9ad google::LogMessage::Fail() at ??:0
@ 0x7f6f1657e88a google::LogMessage::SendToLog() at ??:0
@ 0x7f6f1657b4de google::LogMessage::Flush() at ??:0
@ 0x7f6f1657ca19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f6f20ebafdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f6f20ebbd2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f6f20eb8579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f6f20efc640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f6f187cd3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f6f187cdc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f6f187cf3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f6f187be512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f6f19e026db start_thread at ??:0
@ 0x7f6f1507961f clone at ??:0
W20260808 22:24:30.019542 32628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:24:30.019790 32628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:24:30.019836 32628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:24:30.023904 32628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:24:30.023962 32628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:24:30.024032 32628 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.193
I20260808 22:24:30.029362 32628 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:33087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.193.keytab
--principal=kudu/127.9.176.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.193:34985
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.176.193
--webserver_port=42243
--tserver_master_addrs=127.9.176.254:38597
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32628
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.193
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:24:30.030408 32628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:24:30.031419 32628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:24:30.040771 32682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:24:30.040812 32685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:24:30.040802 32683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:24:30.041148 32628 server_base.cc:1034] running on GCE node
I20260808 22:24:30.041608 32628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:24:30.042400 32628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:24:30.043604 32628 hybrid_clock.cc:648] HybridClock initialized: now 1786227870043579 us; error 33 us; skew 500 ppm
Aug 08 22:24:30 dist-test-slave-ncp9 krb5kdc[30953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227870, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:24:30.046924 32628 init.cc:377] Logged in from keytab as kudu/127.9.176.193@KRBTEST.COM (short username kudu)
I20260808 22:24:30.047912 32628 webserver.cc:460] Webserver started at http://127.9.176.193:42243/ using document root <none> and password file <none>
I20260808 22:24:30.048604 32628 fs_manager.cc:359] Metadata directory not provided
I20260808 22:24:30.048702 32628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:24:30.053697 32628 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 22:24:30.056325 32692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:24:30.057790 32628 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 22:24:30.057914 32628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-0/wal
uuid: "c725f56e88ba4e649a6e854587c1da74"
format_stamp: "Formatted at 2026-08-08 22:24:27 on dist-test-slave-ncp9"
server_key: "c52a2e17aa4900cef485315f7c3cbfe0"
server_key_iv: "abb258a6a786adb289fd93f8bd4a7462"
server_key_version: "encryptionkey@0"
I20260808 22:24:30.058609 32628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:24:30.083056 32628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:24:30.086578 32628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:24:30.086769 32628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:24:30.087546 32628 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:24:30.088843 32700 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260808 22:24:30.092819 32628 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260808 22:24:30.092932 32628 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20260808 22:24:30.093011 32628 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260808 22:24:30.095165 32628 ts_tablet_manager.cc:610] Registered 1 tablets
I20260808 22:24:30.095222 32628 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260808 22:24:30.095556 32700 tablet_bootstrap.cc:492] T c358d840f37b4fc1b57911428d8fb659 P c725f56e88ba4e649a6e854587c1da74: Bootstrap starting.
I20260808 22:24:30.105998 32628 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.193:34985
I20260808 22:24:30.106236 379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.193:34985 every 8 connection(s)
I20260808 22:24:30.107151 32628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
I20260808 22:24:30.111001 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 32628
I20260808 22:24:30.111496 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.194:37751
--local_ip_for_outbound_sockets=127.9.176.194
--tserver_master_addrs=127.9.176.254:38597
--webserver_port=44689
--webserver_interface=127.9.176.194
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.194.keytab
--principal=kudu/127.9.176.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:33087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 22:24:30.115708 32700 log.cc:824] T c358d840f37b4fc1b57911428d8fb659 P c725f56e88ba4e649a6e854587c1da74: Log is configured to *not* fsync() on all Append() calls
I20260808 22:24:30.124475 32700 tablet_bootstrap.cc:492] T c358d840f37b4fc1b57911428d8fb659 P c725f56e88ba4e649a6e854587c1da74: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 22:24:30.124876 32700 tablet_bootstrap.cc:492] T c358d840f37b4fc1b57911428d8fb659 P c725f56e88ba4e649a6e854587c1da74: Bootstrap complete.
I20260808 22:24:30.126428 32700 ts_tablet_manager.cc:1397] T c358d840f37b4fc1b57911428d8fb659 P c725f56e88ba4e649a6e854587c1da74: Time spent bootstrapping tablet: real 0.031s user 0.018s sys 0.011s
I20260808 22:24:30.129700 32700 raft_consensus.cc:348] T c358d840f37b4fc1b57911428d8fb659 P c725f56e88ba4e649a6e854587c1da74 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7519d2ff3a643fdbc922255349cc2d1" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 44949 } } peers { permanent_uuid: "b9bdb3f4a3d84fb2b48be7e89cd5672f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 37751 } } peers { permanent_uuid: "c725f56e88ba4e649a6e854587c1da74" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34985 } }
I20260808 22:24:30.130249 32700 raft_consensus.cc:729] T c358d840f37b4fc1b57911428d8fb659 P c725f56e88ba4e649a6e854587c1da74 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c725f56e88ba4e649a6e854587c1da74, State: Initialized, Role: FOLLOWER
I20260808 22:24:30.130798 32700 consensus_queue.cc:260] T c358d840f37b4fc1b57911428d8fb659 P c725f56e88ba4e649a6e854587c1da74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7519d2ff3a643fdbc922255349cc2d1" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 44949 } } peers { permanent_uuid: "b9bdb3f4a3d84fb2b48be7e89cd5672f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 37751 } } peers { permanent_uuid: "c725f56e88ba4e649a6e854587c1da74" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34985 } }
I20260808 22:24:30.131896 32700 ts_tablet_manager.cc:1428] T c358d840f37b4fc1b57911428d8fb659 P c725f56e88ba4e649a6e854587c1da74: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
W20260808 22:24:30.174026 383 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:24:30.109302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:38597 (local address 127.9.176.193:45753)
0808 22:24:30.109830 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:30.109874 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 22:24:30.110628 (+ 754us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:24:30.173065 (+ 62437us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:38597: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":143,"threads_started":1}
W20260808 22:24:30.174556 380 heartbeater.cc:644] Failed to heartbeat to 127.9.176.254:38597 (0 consecutive failures): Network error: Failed to ping master at 127.9.176.254:38597: Client connection negotiation failed: client connection to 127.9.176.254:38597: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260808 22:24:30.175334 383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:30.175057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:38597 (local address 127.9.176.193:45477)
0808 22:24:30.175195 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:30.175272 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:38597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260808 22:24:30.175990 383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:30.175782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:38597 (local address 127.9.176.193:33871)
0808 22:24:30.175887 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:30.175936 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:38597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260808 22:24:30.176214 380 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 22:24:30.227068 386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:24:30.227309 386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:24:30.227351 386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:24:30.231827 386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:24:30.231881 386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:24:30.231937 386 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.194
I20260808 22:24:30.237120 386 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:33087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.194.keytab
--principal=kudu/127.9.176.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.194:37751
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.176.194
--webserver_port=44689
--tserver_master_addrs=127.9.176.254:38597
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.386
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.194
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:24:30.238090 386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:24:30.239010 386 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:24:30.248162 393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:24:30.248135 401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:24:30.248180 395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:24:30.248425 386 server_base.cc:1034] running on GCE node
I20260808 22:24:30.248801 386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:24:30.249406 386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:24:30.250568 386 hybrid_clock.cc:648] HybridClock initialized: now 1786227870250551 us; error 28 us; skew 500 ppm
Aug 08 22:24:30 dist-test-slave-ncp9 krb5kdc[30953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227870, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:24:30.253176 386 init.cc:377] Logged in from keytab as kudu/127.9.176.194@KRBTEST.COM (short username kudu)
I20260808 22:24:30.253931 386 webserver.cc:460] Webserver started at http://127.9.176.194:44689/ using document root <none> and password file <none>
I20260808 22:24:30.254433 386 fs_manager.cc:359] Metadata directory not provided
I20260808 22:24:30.254491 386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:24:30.258080 386 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260808 22:24:30.259925 412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:24:30.261081 386 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 22:24:30.261170 386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-1/wal
uuid: "b9bdb3f4a3d84fb2b48be7e89cd5672f"
format_stamp: "Formatted at 2026-08-08 22:24:27 on dist-test-slave-ncp9"
server_key: "d2e6403be0a1e99d03f9585ac7abcfdd"
server_key_iv: "2fe1edf73c12e43a6bc9fe61d223548d"
server_key_version: "encryptionkey@0"
I20260808 22:24:30.261656 386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:24:30.294076 386 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:24:30.297055 386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:24:30.297200 386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:24:30.297714 386 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:24:30.298481 386 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:24:30.298525 386 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:24:30.298564 386 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:24:30.298584 386 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:24:30.306917 386 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.194:37751
I20260808 22:24:30.306951 570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.194:37751 every 8 connection(s)
I20260808 22:24:30.307878 386 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
I20260808 22:24:30.310428 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 386
I20260808 22:24:30.310866 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.195:44949
--local_ip_for_outbound_sockets=127.9.176.195
--tserver_master_addrs=127.9.176.254:38597
--webserver_port=39893
--webserver_interface=127.9.176.195
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.195.keytab
--principal=kudu/127.9.176.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:33087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 22:24:30.311775 579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:30.309924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:38597 (local address 127.9.176.194:51851)
0808 22:24:30.310381 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:30.310484 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:38597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":114,"threads_started":1}
W20260808 22:24:30.312318 572 heartbeater.cc:644] Failed to heartbeat to 127.9.176.254:38597 (0 consecutive failures): Network error: Failed to ping master at 127.9.176.254:38597: Client connection negotiation failed: client connection to 127.9.176.254:38597: connect: Connection refused (error 111)
I20260808 22:24:30.312968 579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:30.312722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:38597 (local address 127.9.176.194:56909)
0808 22:24:30.312831 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:30.312879 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:38597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260808 22:24:30.313541 579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:30.313341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:38597 (local address 127.9.176.194:42445)
0808 22:24:30.313442 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:30.313488 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:38597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260808 22:24:30.313688 572 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 22:24:30.419008 580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:24:30.419240 580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:24:30.419297 580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:24:30.423168 580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:24:30.423219 580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:24:30.423292 580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.195
I20260808 22:24:30.428247 580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:33087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.195.keytab
--principal=kudu/127.9.176.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.195:44949
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.176.195
--webserver_port=39893
--tserver_master_addrs=127.9.176.254:38597
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.195
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:24:30.429180 580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:24:30.430117 580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:24:30.439311 590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:24:30.439381 587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:24:30.439345 580 server_base.cc:1034] running on GCE node
W20260808 22:24:30.439312 588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:24:30.439957 580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:24:30.440546 580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:24:30.441704 580 hybrid_clock.cc:648] HybridClock initialized: now 1786227870441684 us; error 30 us; skew 500 ppm
Aug 08 22:24:30 dist-test-slave-ncp9 krb5kdc[30953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227870, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:24:30.444432 580 init.cc:377] Logged in from keytab as kudu/127.9.176.195@KRBTEST.COM (short username kudu)
I20260808 22:24:30.445331 580 webserver.cc:460] Webserver started at http://127.9.176.195:39893/ using document root <none> and password file <none>
I20260808 22:24:30.445803 580 fs_manager.cc:359] Metadata directory not provided
I20260808 22:24:30.445858 580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:24:30.449492 580 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260808 22:24:30.451367 598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:24:30.452464 580 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 22:24:30.452558 580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-2/wal
uuid: "e7519d2ff3a643fdbc922255349cc2d1"
format_stamp: "Formatted at 2026-08-08 22:24:28 on dist-test-slave-ncp9"
server_key: "ac780607a96c5bd1326617b7c3a4b343"
server_key_iv: "c95496a88a66d3ffaa5170a090833e43"
server_key_version: "encryptionkey@0"
I20260808 22:24:30.453083 580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:24:30.466974 580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:24:30.469750 580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:24:30.469919 580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:24:30.470435 580 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:24:30.471199 580 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:24:30.471246 580 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:24:30.471297 580 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:24:30.471324 580 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:24:30.479418 580 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.195:44949
I20260808 22:24:30.479470 759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.195:44949 every 8 connection(s)
I20260808 22:24:30.480337 580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
I20260808 22:24:30.484375 762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:30.482342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:38597 (local address 127.9.176.195:43343)
0808 22:24:30.482791 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:30.482928 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:38597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":124,"threads_started":1}
W20260808 22:24:30.485098 760 heartbeater.cc:644] Failed to heartbeat to 127.9.176.254:38597 (0 consecutive failures): Network error: Failed to ping master at 127.9.176.254:38597: Client connection negotiation failed: client connection to 127.9.176.254:38597: connect: Connection refused (error 111)
I20260808 22:24:30.485755 762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:30.485525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:38597 (local address 127.9.176.195:46607)
0808 22:24:30.485631 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:30.485689 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:38597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260808 22:24:30.486394 762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:30.486211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:38597 (local address 127.9.176.195:54025)
0808 22:24:30.486304 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:30.486345 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:38597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260808 22:24:30.486558 760 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260808 22:24:30.487542 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 580
I20260808 22:24:31.177223 766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:31.176766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:38597 (local address 127.9.176.193:60071)
0808 22:24:31.177060 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:31.177117 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:38597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":75,"threads_started":1}
I20260808 22:24:31.300822 767 raft_consensus.cc:482] T c358d840f37b4fc1b57911428d8fb659 P c725f56e88ba4e649a6e854587c1da74 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 22:24:31.301072 767 raft_consensus.cc:504] T c358d840f37b4fc1b57911428d8fb659 P c725f56e88ba4e649a6e854587c1da74 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7519d2ff3a643fdbc922255349cc2d1" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 44949 } } peers { permanent_uuid: "b9bdb3f4a3d84fb2b48be7e89cd5672f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 37751 } } peers { permanent_uuid: "c725f56e88ba4e649a6e854587c1da74" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34985 } }
I20260808 22:24:31.302115 767 leader_election.cc:288] T c358d840f37b4fc1b57911428d8fb659 P c725f56e88ba4e649a6e854587c1da74 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e7519d2ff3a643fdbc922255349cc2d1 (127.9.176.195:44949), b9bdb3f4a3d84fb2b48be7e89cd5672f (127.9.176.194:37751)
Aug 08 22:24:31 dist-test-slave-ncp9 krb5kdc[30953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227870, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.193@KRBTEST.COM for kudu/127.9.176.195@KRBTEST.COM
Aug 08 22:24:31 dist-test-slave-ncp9 krb5kdc[30953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227870, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.193@KRBTEST.COM for kudu/127.9.176.194@KRBTEST.COM
I20260808 22:24:31.314937 775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:31.314144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:38597 (local address 127.9.176.194:37711)
0808 22:24:31.314777 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:31.314834 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:38597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":72,"threads_started":1}
I20260808 22:24:31.318222 768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:31.302377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.194:37751 (local address 127.9.176.193:53945)
0808 22:24:31.302887 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:31.302900 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 22:24:31.303009 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:24:31.304261 (+ 1252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:24:31.304267 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:24:31.304807 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:24:31.305332 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:24:31.305347 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:24:31.306601 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:24:31.306604 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:24:31.307651 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:24:31.307661 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:24:31.308614 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:24:31.308617 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:24:31.308745 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:31.312081 (+ 3336us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:24:31.312098 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:24:31.313974 (+ 1876us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:24:31.316009 (+ 2035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:24:31.316012 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:24:31.316013 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:24:31.316254 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:24:31.316559 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:24:31.316562 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:24:31.316564 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:24:31.316707 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:24:31.317322 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:24:31.317327 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:24:31.317561 (+ 234us) client_negotiation.cc:769] Sending connection context
0808 22:24:31.317719 (+ 158us) client_negotiation.cc:240] Negotiation successful
0808 22:24:31.318049 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"mutex_wait_us":177,"spinlock_wait_cycles":35968,"thread_start_us":71,"threads_started":1}
I20260808 22:24:31.318262 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:31.302509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.193:41069 (local address 127.9.176.195:44949)
0808 22:24:31.302753 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0808 22:24:31.302770 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 22:24:31.302815 (+ 45us) server_negotiation.cc:407] Connection header received
0808 22:24:31.303548 (+ 733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:24:31.303556 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:24:31.303908 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:24:31.304182 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:24:31.305492 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:31.306457 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:31.307811 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:31.308439 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:31.308603 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:31.313942 (+ 5339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:24:31.313949 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:24:31.313973 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:24:31.314008 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:24:31.315707 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:24:31.316323 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:24:31.316329 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:24:31.316335 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:24:31.316449 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:24:31.316789 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:24:31.316792 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:24:31.316794 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:24:31.317210 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:24:31.317328 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0808 22:24:31.317850 (+ 522us) server_negotiation.cc:299] Negotiation successful
0808 22:24:31.318107 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":59,"threads_started":1}
I20260808 22:24:31.318564 766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:31.302310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.195:44949 (local address 127.9.176.193:41069)
0808 22:24:31.302510 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:31.302525 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 22:24:31.302626 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:24:31.304262 (+ 1636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:24:31.304268 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:24:31.304807 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:24:31.305332 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:24:31.305347 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:24:31.306593 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:24:31.306596 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:24:31.307679 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:24:31.307691 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:24:31.308558 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:24:31.308561 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:24:31.308742 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:31.312094 (+ 3352us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:24:31.312105 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:24:31.313700 (+ 1595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:24:31.315831 (+ 2131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:24:31.315837 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:24:31.315850 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:24:31.316219 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:24:31.316554 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:24:31.316557 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:24:31.316559 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:24:31.316700 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:24:31.317322 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:24:31.317327 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:24:31.317566 (+ 239us) client_negotiation.cc:769] Sending connection context
0808 22:24:31.317720 (+ 154us) client_negotiation.cc:240] Negotiation successful
0808 22:24:31.318126 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"mutex_wait_us":314,"spinlock_wait_cycles":21504}
I20260808 22:24:31.318270 774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:31.302530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.193:53945 (local address 127.9.176.194:37751)
0808 22:24:31.302853 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0808 22:24:31.302863 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 22:24:31.302987 (+ 124us) server_negotiation.cc:407] Connection header received
0808 22:24:31.303676 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:24:31.303681 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:24:31.303976 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:24:31.304226 (+ 250us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:24:31.305497 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:31.306454 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:31.307756 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:31.308453 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:31.308842 (+ 389us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:31.314147 (+ 5305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:24:31.314154 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:24:31.314167 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:24:31.314198 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:24:31.315897 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:24:31.316356 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:24:31.316361 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:24:31.316364 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:24:31.316468 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:24:31.316796 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:24:31.316799 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:24:31.316800 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:24:31.317211 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:24:31.317328 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0808 22:24:31.317854 (+ 526us) server_negotiation.cc:299] Negotiation successful
0808 22:24:31.318100 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":83,"threads_started":1}
W20260808 22:24:31.320834 32696 leader_election.cc:341] T c358d840f37b4fc1b57911428d8fb659 P c725f56e88ba4e649a6e854587c1da74 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e7519d2ff3a643fdbc922255349cc2d1 (127.9.176.195:44949): Not found: Tablet not found: c358d840f37b4fc1b57911428d8fb659
W20260808 22:24:31.321009 32693 leader_election.cc:341] T c358d840f37b4fc1b57911428d8fb659 P c725f56e88ba4e649a6e854587c1da74 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b9bdb3f4a3d84fb2b48be7e89cd5672f (127.9.176.194:37751): Not found: Tablet not found: c358d840f37b4fc1b57911428d8fb659
I20260808 22:24:31.321081 32693 leader_election.cc:302] T c358d840f37b4fc1b57911428d8fb659 P c725f56e88ba4e649a6e854587c1da74 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c725f56e88ba4e649a6e854587c1da74; no voters: b9bdb3f4a3d84fb2b48be7e89cd5672f, e7519d2ff3a643fdbc922255349cc2d1
I20260808 22:24:31.321307 767 raft_consensus.cc:2726] T c358d840f37b4fc1b57911428d8fb659 P c725f56e88ba4e649a6e854587c1da74 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 22:24:31.487486 776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:31.487058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:38597 (local address 127.9.176.195:45411)
0808 22:24:31.487315 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:31.487375 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:38597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":96,"threads_started":1}
I20260808 22:24:32.178306 777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:32.177917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:38597 (local address 127.9.176.193:41699)
0808 22:24:32.178148 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:32.178204 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:38597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":86,"threads_started":1}
I20260808 22:24:32.316079 780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:32.315682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:38597 (local address 127.9.176.194:52677)
0808 22:24:32.315922 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:32.315978 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:38597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":73,"threads_started":1}
I20260808 22:24:32.488677 781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:32.488259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:38597 (local address 127.9.176.195:35861)
0808 22:24:32.488527 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:32.488578 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:38597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":78,"threads_started":1}
I20260808 22:24:32.968461 782 raft_consensus.cc:482] T c358d840f37b4fc1b57911428d8fb659 P c725f56e88ba4e649a6e854587c1da74 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 22:24:32.968592 782 raft_consensus.cc:504] T c358d840f37b4fc1b57911428d8fb659 P c725f56e88ba4e649a6e854587c1da74 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7519d2ff3a643fdbc922255349cc2d1" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 44949 } } peers { permanent_uuid: "b9bdb3f4a3d84fb2b48be7e89cd5672f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 37751 } } peers { permanent_uuid: "c725f56e88ba4e649a6e854587c1da74" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34985 } }
I20260808 22:24:32.969033 782 leader_election.cc:288] T c358d840f37b4fc1b57911428d8fb659 P c725f56e88ba4e649a6e854587c1da74 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e7519d2ff3a643fdbc922255349cc2d1 (127.9.176.195:44949), b9bdb3f4a3d84fb2b48be7e89cd5672f (127.9.176.194:37751)
W20260808 22:24:32.969888 32696 leader_election.cc:341] T c358d840f37b4fc1b57911428d8fb659 P c725f56e88ba4e649a6e854587c1da74 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e7519d2ff3a643fdbc922255349cc2d1 (127.9.176.195:44949): Not found: Tablet not found: c358d840f37b4fc1b57911428d8fb659
W20260808 22:24:32.970011 32693 leader_election.cc:341] T c358d840f37b4fc1b57911428d8fb659 P c725f56e88ba4e649a6e854587c1da74 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b9bdb3f4a3d84fb2b48be7e89cd5672f (127.9.176.194:37751): Not found: Tablet not found: c358d840f37b4fc1b57911428d8fb659
I20260808 22:24:32.970077 32693 leader_election.cc:302] T c358d840f37b4fc1b57911428d8fb659 P c725f56e88ba4e649a6e854587c1da74 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c725f56e88ba4e649a6e854587c1da74; no voters: b9bdb3f4a3d84fb2b48be7e89cd5672f, e7519d2ff3a643fdbc922255349cc2d1
I20260808 22:24:32.970202 782 raft_consensus.cc:2726] T c358d840f37b4fc1b57911428d8fb659 P c725f56e88ba4e649a6e854587c1da74 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 22:24:33.179499 783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:33.179066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:38597 (local address 127.9.176.193:59677)
0808 22:24:33.179329 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:33.179396 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:38597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":73,"threads_started":1}
I20260808 22:24:33.317152 785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:33.316749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:38597 (local address 127.9.176.194:60603)
0808 22:24:33.316996 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:33.317050 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:38597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":69,"threads_started":1}
I20260808 22:24:33.489710 787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:33.489307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:38597 (local address 127.9.176.195:45723)
0808 22:24:33.489562 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:33.489625 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:38597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":80,"threads_started":1}
I20260808 22:24:34.180557 789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:34.180168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:38597 (local address 127.9.176.193:41307)
0808 22:24:34.180402 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:34.180453 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:38597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":75,"threads_started":1}
I20260808 22:24:34.318166 790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:34.317787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:38597 (local address 127.9.176.194:32811)
0808 22:24:34.318020 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:34.318075 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:38597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":66,"threads_started":1}
I20260808 22:24:34.490816 792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:34.490387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:38597 (local address 127.9.176.195:57735)
0808 22:24:34.490663 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:34.490727 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:38597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":84,"threads_started":1}
I20260808 22:24:35.181913 794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:35.181449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:38597 (local address 127.9.176.193:45463)
0808 22:24:35.181757 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:35.181815 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:38597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":79,"threads_started":1}
I20260808 22:24:35.319319 795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:35.318928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:38597 (local address 127.9.176.194:60089)
0808 22:24:35.319168 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:35.319223 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:38597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":73,"threads_started":1}
I20260808 22:24:35.491868 797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:35.491504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:38597 (local address 127.9.176.195:43945)
0808 22:24:35.491731 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:35.491781 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:38597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":70,"threads_started":1}
I20260808 22:24:35.538412 800 raft_consensus.cc:482] T c358d840f37b4fc1b57911428d8fb659 P c725f56e88ba4e649a6e854587c1da74 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 22:24:35.538546 800 raft_consensus.cc:504] T c358d840f37b4fc1b57911428d8fb659 P c725f56e88ba4e649a6e854587c1da74 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7519d2ff3a643fdbc922255349cc2d1" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 44949 } } peers { permanent_uuid: "b9bdb3f4a3d84fb2b48be7e89cd5672f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 37751 } } peers { permanent_uuid: "c725f56e88ba4e649a6e854587c1da74" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34985 } }
I20260808 22:24:35.538918 800 leader_election.cc:288] T c358d840f37b4fc1b57911428d8fb659 P c725f56e88ba4e649a6e854587c1da74 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e7519d2ff3a643fdbc922255349cc2d1 (127.9.176.195:44949), b9bdb3f4a3d84fb2b48be7e89cd5672f (127.9.176.194:37751)
W20260808 22:24:35.539702 32696 leader_election.cc:341] T c358d840f37b4fc1b57911428d8fb659 P c725f56e88ba4e649a6e854587c1da74 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e7519d2ff3a643fdbc922255349cc2d1 (127.9.176.195:44949): Not found: Tablet not found: c358d840f37b4fc1b57911428d8fb659
W20260808 22:24:35.539824 32693 leader_election.cc:341] T c358d840f37b4fc1b57911428d8fb659 P c725f56e88ba4e649a6e854587c1da74 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b9bdb3f4a3d84fb2b48be7e89cd5672f (127.9.176.194:37751): Not found: Tablet not found: c358d840f37b4fc1b57911428d8fb659
I20260808 22:24:35.539892 32693 leader_election.cc:302] T c358d840f37b4fc1b57911428d8fb659 P c725f56e88ba4e649a6e854587c1da74 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c725f56e88ba4e649a6e854587c1da74; no voters: b9bdb3f4a3d84fb2b48be7e89cd5672f, e7519d2ff3a643fdbc922255349cc2d1
I20260808 22:24:35.540027 800 raft_consensus.cc:2726] T c358d840f37b4fc1b57911428d8fb659 P c725f56e88ba4e649a6e854587c1da74 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 22:24:36.183161 801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:36.182722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:38597 (local address 127.9.176.193:59541)
0808 22:24:36.183004 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:36.183059 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:38597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":80,"threads_started":1}
I20260808 22:24:36.320466 802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:36.320044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:38597 (local address 127.9.176.194:35455)
0808 22:24:36.320314 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:36.320372 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:38597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":80,"threads_started":1}
I20260808 22:24:36.493046 804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:36.492598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:38597 (local address 127.9.176.195:56561)
0808 22:24:36.492848 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:36.492946 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:38597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":73,"threads_started":1}
I20260808 22:24:37.184370 805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:37.183919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:38597 (local address 127.9.176.193:49399)
0808 22:24:37.184198 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:37.184268 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:38597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":88,"threads_started":1}
I20260808 22:24:37.321627 807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:37.321180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:38597 (local address 127.9.176.194:43209)
0808 22:24:37.321472 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:37.321532 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:38597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":70,"threads_started":1}
I20260808 22:24:37.494334 808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:37.493821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:38597 (local address 127.9.176.195:54527)
0808 22:24:37.494156 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:37.494216 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:38597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":71,"threads_started":1}
I20260808 22:24:38.185509 809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:38.185118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:38597 (local address 127.9.176.193:39213)
0808 22:24:38.185358 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:38.185412 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:38597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":76,"threads_started":1}
I20260808 22:24:38.322734 811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:38.322318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:38597 (local address 127.9.176.194:34079)
0808 22:24:38.322587 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:38.322643 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:38597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":75,"threads_started":1}
I20260808 22:24:38.495536 813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:38.495104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:38597 (local address 127.9.176.195:56149)
0808 22:24:38.495363 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:38.495420 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:38597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":82,"threads_started":1}
I20260808 22:24:38.919579 814 raft_consensus.cc:482] T c358d840f37b4fc1b57911428d8fb659 P c725f56e88ba4e649a6e854587c1da74 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 22:24:38.919740 814 raft_consensus.cc:504] T c358d840f37b4fc1b57911428d8fb659 P c725f56e88ba4e649a6e854587c1da74 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7519d2ff3a643fdbc922255349cc2d1" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 44949 } } peers { permanent_uuid: "b9bdb3f4a3d84fb2b48be7e89cd5672f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 37751 } } peers { permanent_uuid: "c725f56e88ba4e649a6e854587c1da74" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34985 } }
I20260808 22:24:38.920217 814 leader_election.cc:288] T c358d840f37b4fc1b57911428d8fb659 P c725f56e88ba4e649a6e854587c1da74 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e7519d2ff3a643fdbc922255349cc2d1 (127.9.176.195:44949), b9bdb3f4a3d84fb2b48be7e89cd5672f (127.9.176.194:37751)
W20260808 22:24:38.920967 32696 leader_election.cc:341] T c358d840f37b4fc1b57911428d8fb659 P c725f56e88ba4e649a6e854587c1da74 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e7519d2ff3a643fdbc922255349cc2d1 (127.9.176.195:44949): Not found: Tablet not found: c358d840f37b4fc1b57911428d8fb659
W20260808 22:24:38.921092 32693 leader_election.cc:341] T c358d840f37b4fc1b57911428d8fb659 P c725f56e88ba4e649a6e854587c1da74 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b9bdb3f4a3d84fb2b48be7e89cd5672f (127.9.176.194:37751): Not found: Tablet not found: c358d840f37b4fc1b57911428d8fb659
I20260808 22:24:38.921177 32693 leader_election.cc:302] T c358d840f37b4fc1b57911428d8fb659 P c725f56e88ba4e649a6e854587c1da74 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c725f56e88ba4e649a6e854587c1da74; no voters: b9bdb3f4a3d84fb2b48be7e89cd5672f, e7519d2ff3a643fdbc922255349cc2d1
I20260808 22:24:38.921336 814 raft_consensus.cc:2726] T c358d840f37b4fc1b57911428d8fb659 P c725f56e88ba4e649a6e854587c1da74 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 22:24:39.186708 815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:39.186271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:38597 (local address 127.9.176.193:44775)
0808 22:24:39.186547 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:39.186607 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:38597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":75,"threads_started":1}
I20260808 22:24:39.323876 816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:39.323447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:38597 (local address 127.9.176.194:54811)
0808 22:24:39.323724 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:39.323781 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:38597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":94,"threads_started":1}
I20260808 22:24:39.496773 817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:39.496255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:38597 (local address 127.9.176.195:53033)
0808 22:24:39.496601 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:39.496669 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:38597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":97,"threads_started":1}
I20260808 22:24:40.188010 818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:40.187602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:38597 (local address 127.9.176.193:53759)
0808 22:24:40.187854 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:40.187910 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:38597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":79,"threads_started":1}
I20260808 22:24:40.325032 819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:40.324613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:38597 (local address 127.9.176.194:38873)
0808 22:24:40.324852 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:40.324933 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:38597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":77,"threads_started":1}
I20260808 22:24:40.497911 821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:40.497528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:38597 (local address 127.9.176.195:41799)
0808 22:24:40.497769 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:40.497823 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:38597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":77,"threads_started":1}
I20260808 22:24:41.189191 824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:41.188736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:38597 (local address 127.9.176.193:49515)
0808 22:24:41.189037 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:41.189094 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:38597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":95,"threads_started":1}
I20260808 22:24:41.326040 825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:41.325662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:38597 (local address 127.9.176.194:54407)
0808 22:24:41.325893 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:41.325947 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:38597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":70,"threads_started":1}
I20260808 22:24:41.498948 826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:41.498556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:38597 (local address 127.9.176.195:54359)
0808 22:24:41.498800 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:41.498855 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:38597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":80,"threads_started":1}
I20260808 22:24:42.190302 827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:42.189896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:38597 (local address 127.9.176.193:55515)
0808 22:24:42.190152 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:42.190205 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:38597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":77,"threads_started":1}
I20260808 22:24:42.327113 828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:42.326732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:38597 (local address 127.9.176.194:52075)
0808 22:24:42.326958 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:42.327013 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:38597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":75,"threads_started":1}
I20260808 22:24:42.499996 830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:42.499575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:38597 (local address 127.9.176.195:46699)
0808 22:24:42.499845 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:42.499900 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:38597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":93,"threads_started":1}
I20260808 22:24:42.994953 831 raft_consensus.cc:482] T c358d840f37b4fc1b57911428d8fb659 P c725f56e88ba4e649a6e854587c1da74 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 22:24:42.995085 831 raft_consensus.cc:504] T c358d840f37b4fc1b57911428d8fb659 P c725f56e88ba4e649a6e854587c1da74 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7519d2ff3a643fdbc922255349cc2d1" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 44949 } } peers { permanent_uuid: "b9bdb3f4a3d84fb2b48be7e89cd5672f" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 37751 } } peers { permanent_uuid: "c725f56e88ba4e649a6e854587c1da74" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34985 } }
I20260808 22:24:42.995482 831 leader_election.cc:288] T c358d840f37b4fc1b57911428d8fb659 P c725f56e88ba4e649a6e854587c1da74 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e7519d2ff3a643fdbc922255349cc2d1 (127.9.176.195:44949), b9bdb3f4a3d84fb2b48be7e89cd5672f (127.9.176.194:37751)
W20260808 22:24:42.996243 32696 leader_election.cc:341] T c358d840f37b4fc1b57911428d8fb659 P c725f56e88ba4e649a6e854587c1da74 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e7519d2ff3a643fdbc922255349cc2d1 (127.9.176.195:44949): Not found: Tablet not found: c358d840f37b4fc1b57911428d8fb659
W20260808 22:24:42.996366 32693 leader_election.cc:341] T c358d840f37b4fc1b57911428d8fb659 P c725f56e88ba4e649a6e854587c1da74 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b9bdb3f4a3d84fb2b48be7e89cd5672f (127.9.176.194:37751): Not found: Tablet not found: c358d840f37b4fc1b57911428d8fb659
I20260808 22:24:42.996467 32693 leader_election.cc:302] T c358d840f37b4fc1b57911428d8fb659 P c725f56e88ba4e649a6e854587c1da74 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c725f56e88ba4e649a6e854587c1da74; no voters: b9bdb3f4a3d84fb2b48be7e89cd5672f, e7519d2ff3a643fdbc922255349cc2d1
I20260808 22:24:42.996604 831 raft_consensus.cc:2726] T c358d840f37b4fc1b57911428d8fb659 P c725f56e88ba4e649a6e854587c1da74 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 22:24:43.191421 832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:43.191004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:38597 (local address 127.9.176.193:59893)
0808 22:24:43.191275 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:43.191327 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:38597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":89,"threads_started":1}
I20260808 22:24:43.328166 834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:43.327759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:38597 (local address 127.9.176.194:52953)
0808 22:24:43.328017 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:43.328072 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:38597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":75,"threads_started":1}
I20260808 22:24:43.501048 835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:43.500667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:38597 (local address 127.9.176.195:50103)
0808 22:24:43.500882 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:43.500957 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:38597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":73,"threads_started":1}
I20260808 22:24:44.192595 836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:44.192116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:38597 (local address 127.9.176.193:39893)
0808 22:24:44.192423 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:44.192490 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:38597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":85,"threads_started":1}
I20260808 22:24:44.329407 837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:44.328930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:38597 (local address 127.9.176.194:49013)
0808 22:24:44.329250 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:44.329305 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:38597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":112,"threads_started":1}
I20260808 22:24:44.502168 841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:44.501743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:38597 (local address 127.9.176.195:59573)
0808 22:24:44.502009 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:44.502071 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:38597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":81,"threads_started":1}
I20260808 22:24:45.193820 842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:45.193457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:38597 (local address 127.9.176.193:35197)
0808 22:24:45.193677 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:45.193725 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:38597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":73,"threads_started":1}
I20260808 22:24:45.330701 844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:45.330294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:38597 (local address 127.9.176.194:52011)
0808 22:24:45.330545 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:45.330598 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:38597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":82,"threads_started":1}
I20260808 22:24:45.490231 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 32628
I20260808 22:24:45.502799 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 386
I20260808 22:24:45.504668 855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:45.503054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:38597 (local address 127.9.176.195:47535)
0808 22:24:45.504508 (+ 1454us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:45.504571 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:38597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1368,"thread_start_us":65,"threads_started":1}
I20260808 22:24:45.510887 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 580
2026-08-08T22:24:45Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19467 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 22:24:45 dist-test-slave-ncp9 krb5kdc[860](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 22:24:45 dist-test-slave-ncp9 krb5kdc[860](info): set up 2 sockets
Aug 08 22:24:45 dist-test-slave-ncp9 krb5kdc[860](info): commencing operation
krb5kdc: starting...
W20260808 22:24:46.641800 9923 mini_kdc.cc:121] Time spent starting KDC: real 1.105s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/krb5kdc/test-user.keytab.
Aug 08 22:24:46 dist-test-slave-ncp9 krb5kdc[860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T22:24:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:24:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.176.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.176.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.176.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab.
Entry for principal kudu/127.9.176.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab.
I20260808 22:24:46.765007 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:39769
--webserver_interface=127.9.176.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab
--principal=kudu/127.9.176.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:33147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.176.254:39769
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:24:46.875835 878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:24:46.876089 878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:24:46.876128 878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:24:46.880021 878 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260808 22:24:46.880079 878 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260808 22:24:46.880093 878 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260808 22:24:46.880105 878 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:24:46.880126 878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:24:46.880148 878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:24:46.880162 878 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:24:46.880177 878 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:24:46.885403 878 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:33147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.9.176.254:39769
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab
--principal=kudu/127.9.176.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:39769
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.176.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.878
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:24:46.886406 878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:24:46.887416 878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:24:46.896422 887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:24:46.896479 888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:24:46.896449 878 server_base.cc:1034] running on GCE node
W20260808 22:24:46.896457 893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:24:46.897289 878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:24:46.898365 878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:24:46.899519 878 hybrid_clock.cc:648] HybridClock initialized: now 1786227886899490 us; error 43 us; skew 500 ppm
Aug 08 22:24:46 dist-test-slave-ncp9 krb5kdc[860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227886, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:24:46.906088 878 init.cc:377] Logged in from keytab as kudu/127.9.176.254@KRBTEST.COM (short username kudu)
I20260808 22:24:46.907011 878 webserver.cc:460] Webserver started at http://127.9.176.254:39755/ using document root <none> and password file <none>
I20260808 22:24:46.907543 878 fs_manager.cc:359] Metadata directory not provided
I20260808 22:24:46.907604 878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:24:46.907805 878 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:24:46.909896 878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/master-0/data/instance:
uuid: "d2f7a8bd3d904d54b493540ee9062e2b"
format_stamp: "Formatted at 2026-08-08 22:24:46 on dist-test-slave-ncp9"
server_key: "bf076ba4a10ac6e06aecea6b2951bf93"
server_key_iv: "416756c26c90b7cc146d954b67ace62e"
server_key_version: "encryptionkey@0"
I20260808 22:24:46.910472 878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/master-0/wal/instance:
uuid: "d2f7a8bd3d904d54b493540ee9062e2b"
format_stamp: "Formatted at 2026-08-08 22:24:46 on dist-test-slave-ncp9"
server_key: "bf076ba4a10ac6e06aecea6b2951bf93"
server_key_iv: "416756c26c90b7cc146d954b67ace62e"
server_key_version: "encryptionkey@0"
I20260808 22:24:46.914083 878 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 22:24:46.916344 901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:24:46.917544 878 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 22:24:46.917662 878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/master-0/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/master-0/wal
uuid: "d2f7a8bd3d904d54b493540ee9062e2b"
format_stamp: "Formatted at 2026-08-08 22:24:46 on dist-test-slave-ncp9"
server_key: "bf076ba4a10ac6e06aecea6b2951bf93"
server_key_iv: "416756c26c90b7cc146d954b67ace62e"
server_key_version: "encryptionkey@0"
I20260808 22:24:46.917768 878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:24:46.931809 878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:24:46.934909 878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:24:46.935089 878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:24:46.941852 878 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.254:39769
I20260808 22:24:46.942111 981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.254:39769 every 8 connection(s)
I20260808 22:24:46.942796 878 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
I20260808 22:24:46.945528 983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:24:46.950937 983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2f7a8bd3d904d54b493540ee9062e2b: Bootstrap starting.
I20260808 22:24:46.951678 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 878
I20260808 22:24:46.951800 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/master-0/wal/instance
I20260808 22:24:46.952126 9923 external_mini_cluster.cc:1428] Setting key 952d418e8b20ecca40c6c041037b95b9
I20260808 22:24:46.953826 983 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2f7a8bd3d904d54b493540ee9062e2b: Neither blocks nor log segments found. Creating new log.
I20260808 22:24:46.954537 983 log.cc:824] T 00000000000000000000000000000000 P d2f7a8bd3d904d54b493540ee9062e2b: Log is configured to *not* fsync() on all Append() calls
I20260808 22:24:46.956646 983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2f7a8bd3d904d54b493540ee9062e2b: No bootstrap required, opened a new log
Aug 08 22:24:46 dist-test-slave-ncp9 krb5kdc[860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
I20260808 22:24:46.960868 983 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2f7a8bd3d904d54b493540ee9062e2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2f7a8bd3d904d54b493540ee9062e2b" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 39769 } }
I20260808 22:24:46.961180 983 raft_consensus.cc:374] T 00000000000000000000000000000000 P d2f7a8bd3d904d54b493540ee9062e2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:24:46.961243 983 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2f7a8bd3d904d54b493540ee9062e2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2f7a8bd3d904d54b493540ee9062e2b, State: Initialized, Role: FOLLOWER
I20260808 22:24:46.961678 983 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2f7a8bd3d904d54b493540ee9062e2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2f7a8bd3d904d54b493540ee9062e2b" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 39769 } }
I20260808 22:24:46.961803 983 raft_consensus.cc:388] T 00000000000000000000000000000000 P d2f7a8bd3d904d54b493540ee9062e2b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:24:46.961863 983 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2f7a8bd3d904d54b493540ee9062e2b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:24:46.962002 983 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2f7a8bd3d904d54b493540ee9062e2b [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:24:46.962848 983 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2f7a8bd3d904d54b493540ee9062e2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2f7a8bd3d904d54b493540ee9062e2b" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 39769 } }
I20260808 22:24:46.963171 983 leader_election.cc:302] T 00000000000000000000000000000000 P d2f7a8bd3d904d54b493540ee9062e2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2f7a8bd3d904d54b493540ee9062e2b; no voters:
I20260808 22:24:46.963460 983 leader_election.cc:288] T 00000000000000000000000000000000 P d2f7a8bd3d904d54b493540ee9062e2b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:24:46.963546 988 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2f7a8bd3d904d54b493540ee9062e2b [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:24:46.963727 988 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2f7a8bd3d904d54b493540ee9062e2b [term 1 LEADER]: Becoming Leader. State: Replica: d2f7a8bd3d904d54b493540ee9062e2b, State: Running, Role: LEADER
I20260808 22:24:46.964066 988 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2f7a8bd3d904d54b493540ee9062e2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2f7a8bd3d904d54b493540ee9062e2b" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 39769 } }
I20260808 22:24:46.964742 983 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2f7a8bd3d904d54b493540ee9062e2b [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:24:46.965281 988 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2f7a8bd3d904d54b493540ee9062e2b [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2f7a8bd3d904d54b493540ee9062e2b. Latest consensus state: current_term: 1 leader_uuid: "d2f7a8bd3d904d54b493540ee9062e2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2f7a8bd3d904d54b493540ee9062e2b" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 39769 } } }
I20260808 22:24:46.965481 988 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2f7a8bd3d904d54b493540ee9062e2b [sys.catalog]: This master's current role is: LEADER
I20260808 22:24:46.965680 991 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2f7a8bd3d904d54b493540ee9062e2b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d2f7a8bd3d904d54b493540ee9062e2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2f7a8bd3d904d54b493540ee9062e2b" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 39769 } } }
I20260808 22:24:46.965785 991 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2f7a8bd3d904d54b493540ee9062e2b [sys.catalog]: This master's current role is: LEADER
I20260808 22:24:46.966038 999 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:24:46.966799 986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:46.953360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38268 (local address 127.9.176.254:39769)
0808 22:24:46.953745 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0808 22:24:46.953760 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 22:24:46.953795 (+ 35us) server_negotiation.cc:407] Connection header received
0808 22:24:46.954431 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:24:46.954448 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:24:46.954791 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:24:46.955130 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:24:46.955929 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:46.957010 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:46.958154 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:46.958614 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:46.958725 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:46.961438 (+ 2713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:24:46.961443 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:24:46.961457 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:24:46.961490 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:24:46.963206 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:24:46.963654 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:24:46.963660 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:24:46.963666 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:24:46.963816 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:24:46.964107 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:24:46.964111 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:24:46.964112 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:24:46.964533 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:24:46.964739 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0808 22:24:46.965425 (+ 686us) server_negotiation.cc:299] Negotiation successful
0808 22:24:46.965736 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":102,"threads_started":1}
I20260808 22:24:46.969264 999 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:24:46.974617 999 catalog_manager.cc:1347] Generated new cluster ID: df74ca17b0754fd1913c4823f612329f
I20260808 22:24:46.974678 999 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:24:46.992085 999 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:24:46.993065 999 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:24:47.001499 999 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d2f7a8bd3d904d54b493540ee9062e2b: Generated new TSK 0
I20260808 22:24:47.002249 999 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.176.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.176.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.176.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.193.keytab.
Entry for principal kudu/127.9.176.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.193.keytab.
I20260808 22:24:47.043608 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.193:0
--local_ip_for_outbound_sockets=127.9.176.193
--webserver_interface=127.9.176.193
--webserver_port=0
--tserver_master_addrs=127.9.176.254:39769
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.193.keytab
--principal=kudu/127.9.176.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:33147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:24:47.185092 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:24:47.185370 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:24:47.185412 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:24:47.189667 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:24:47.189735 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:24:47.189795 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.193
I20260808 22:24:47.195062 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:33147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.193.keytab
--principal=kudu/127.9.176.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.176.193
--webserver_port=0
--tserver_master_addrs=127.9.176.254:39769
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.193
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:24:47.196095 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:24:47.197111 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:24:47.206578 1035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:24:47.206612 1033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:24:47.206753 1024 server_base.cc:1034] running on GCE node
W20260808 22:24:47.206668 1032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:24:47.207186 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:24:47.207813 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:24:47.208953 1024 hybrid_clock.cc:648] HybridClock initialized: now 1786227887208931 us; error 33 us; skew 500 ppm
Aug 08 22:24:47 dist-test-slave-ncp9 krb5kdc[860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227887, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:24:47.211609 1024 init.cc:377] Logged in from keytab as kudu/127.9.176.193@KRBTEST.COM (short username kudu)
I20260808 22:24:47.212458 1024 webserver.cc:460] Webserver started at http://127.9.176.193:43873/ using document root <none> and password file <none>
I20260808 22:24:47.213017 1024 fs_manager.cc:359] Metadata directory not provided
I20260808 22:24:47.213088 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:24:47.213303 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:24:47.215293 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-0/data/instance:
uuid: "75595c6cc3d54995ade109df7bbe8820"
format_stamp: "Formatted at 2026-08-08 22:24:47 on dist-test-slave-ncp9"
server_key: "fe802af7e6c84377e23aff40cce8d91c"
server_key_iv: "b670f24ecf1cfa8513b49cdb2376de4a"
server_key_version: "encryptionkey@0"
I20260808 22:24:47.215847 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-0/wal/instance:
uuid: "75595c6cc3d54995ade109df7bbe8820"
format_stamp: "Formatted at 2026-08-08 22:24:47 on dist-test-slave-ncp9"
server_key: "fe802af7e6c84377e23aff40cce8d91c"
server_key_iv: "b670f24ecf1cfa8513b49cdb2376de4a"
server_key_version: "encryptionkey@0"
I20260808 22:24:47.219503 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260808 22:24:47.221931 1046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:24:47.223168 1024 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 22:24:47.223280 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-0/wal
uuid: "75595c6cc3d54995ade109df7bbe8820"
format_stamp: "Formatted at 2026-08-08 22:24:47 on dist-test-slave-ncp9"
server_key: "fe802af7e6c84377e23aff40cce8d91c"
server_key_iv: "b670f24ecf1cfa8513b49cdb2376de4a"
server_key_version: "encryptionkey@0"
I20260808 22:24:47.223378 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:24:47.244463 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:24:47.247548 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:24:47.247740 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:24:47.248355 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:24:47.249243 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:24:47.249292 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:24:47.249346 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:24:47.249370 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:24:47.258183 1024 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.193:34743
I20260808 22:24:47.258213 1221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.193:34743 every 8 connection(s)
I20260808 22:24:47.259049 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
I20260808 22:24:47.261742 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 1024
I20260808 22:24:47.261871 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-0/wal/instance
I20260808 22:24:47.262152 9923 external_mini_cluster.cc:1428] Setting key d4aa00ddcce2695dc810d56ae6c2f336
Aug 08 22:24:47 dist-test-slave-ncp9 krb5kdc[860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227887, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.193@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.176.194@KRBTEST.COM; defaulting to no policy
I20260808 22:24:47.272622 986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:47.260758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.193:50303 (local address 127.9.176.254:39769)
0808 22:24:47.260881 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0808 22:24:47.260885 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:24:47.261735 (+ 850us) server_negotiation.cc:407] Connection header received
0808 22:24:47.263242 (+ 1507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:24:47.263245 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:24:47.263300 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:24:47.263382 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:24:47.264559 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:47.265147 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:47.266382 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:47.266667 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:47.266755 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:47.269328 (+ 2573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:24:47.269331 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:24:47.269333 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:24:47.269357 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:24:47.270646 (+ 1289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:24:47.271128 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:24:47.271131 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:24:47.271132 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:24:47.271212 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:24:47.271525 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:24:47.271528 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:24:47.271530 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:24:47.271718 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:24:47.271807 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0808 22:24:47.272327 (+ 520us) server_negotiation.cc:299] Negotiation successful
0808 22:24:47.272454 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260808 22:24:47.273474 1225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:47.261126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:39769 (local address 127.9.176.193:50303)
0808 22:24:47.261586 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:47.261632 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0808 22:24:47.263037 (+ 1405us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:24:47.263561 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:24:47.263568 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:24:47.263929 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:24:47.264409 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:24:47.264423 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:24:47.265269 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:24:47.265272 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:24:47.266267 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:24:47.266277 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:24:47.266809 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:24:47.266814 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:24:47.266972 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:47.267531 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:24:47.267546 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:24:47.269173 (+ 1627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:24:47.270765 (+ 1592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:24:47.270771 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:24:47.270786 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:24:47.271033 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:24:47.271296 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:24:47.271299 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:24:47.271301 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:24:47.271437 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:24:47.271822 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:24:47.271829 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:24:47.272086 (+ 257us) client_negotiation.cc:769] Sending connection context
0808 22:24:47.272263 (+ 177us) client_negotiation.cc:240] Negotiation successful
0808 22:24:47.272583 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":119,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.176.194@KRBTEST.COM" created.
I20260808 22:24:47.274799 1222 heartbeater.cc:347] Connected to a master server at 127.9.176.254:39769
I20260808 22:24:47.275064 1222 heartbeater.cc:464] Registering TS with master...
I20260808 22:24:47.275681 1222 heartbeater.cc:511] Master 127.9.176.254:39769 requested a full tablet report, sending...
I20260808 22:24:47.277459 926 ts_manager.cc:194] Registered new tserver with Master: 75595c6cc3d54995ade109df7bbe8820 (127.9.176.193:34743)
I20260808 22:24:47.278553 926 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.176.193@KRBTEST.COM'} at 127.9.176.193:50303
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.176.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.194.keytab.
Entry for principal kudu/127.9.176.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.194.keytab.
I20260808 22:24:47.289255 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.194:0
--local_ip_for_outbound_sockets=127.9.176.194
--webserver_interface=127.9.176.194
--webserver_port=0
--tserver_master_addrs=127.9.176.254:39769
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.194.keytab
--principal=kudu/127.9.176.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:33147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:24:47.397701 1229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:24:47.397960 1229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:24:47.398010 1229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:24:47.401921 1229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:24:47.401969 1229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:24:47.402029 1229 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.194
I20260808 22:24:47.407100 1229 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:33147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.194.keytab
--principal=kudu/127.9.176.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.176.194
--webserver_port=0
--tserver_master_addrs=127.9.176.254:39769
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1229
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.194
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:24:47.408068 1229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:24:47.409044 1229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:24:47.418237 1236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:24:47.418324 1229 server_base.cc:1034] running on GCE node
W20260808 22:24:47.418298 1238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:24:47.418237 1235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:24:47.418975 1229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:24:47.419562 1229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:24:47.420718 1229 hybrid_clock.cc:648] HybridClock initialized: now 1786227887420697 us; error 32 us; skew 500 ppm
Aug 08 22:24:47 dist-test-slave-ncp9 krb5kdc[860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227887, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:24:47.423450 1229 init.cc:377] Logged in from keytab as kudu/127.9.176.194@KRBTEST.COM (short username kudu)
I20260808 22:24:47.424319 1229 webserver.cc:460] Webserver started at http://127.9.176.194:44931/ using document root <none> and password file <none>
I20260808 22:24:47.424860 1229 fs_manager.cc:359] Metadata directory not provided
I20260808 22:24:47.424959 1229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:24:47.425172 1229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:24:47.427219 1229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-1/data/instance:
uuid: "87604ec3fe8c43b688447d96bd4bfd3e"
format_stamp: "Formatted at 2026-08-08 22:24:47 on dist-test-slave-ncp9"
server_key: "4a57b74ff8dccf91150efac3fb9918d9"
server_key_iv: "2eb1172c80a8d90234e8baa2c54d6673"
server_key_version: "encryptionkey@0"
I20260808 22:24:47.427767 1229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-1/wal/instance:
uuid: "87604ec3fe8c43b688447d96bd4bfd3e"
format_stamp: "Formatted at 2026-08-08 22:24:47 on dist-test-slave-ncp9"
server_key: "4a57b74ff8dccf91150efac3fb9918d9"
server_key_iv: "2eb1172c80a8d90234e8baa2c54d6673"
server_key_version: "encryptionkey@0"
I20260808 22:24:47.431349 1229 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260808 22:24:47.433621 1249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:24:47.434755 1229 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 22:24:47.434876 1229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-1/wal
uuid: "87604ec3fe8c43b688447d96bd4bfd3e"
format_stamp: "Formatted at 2026-08-08 22:24:47 on dist-test-slave-ncp9"
server_key: "4a57b74ff8dccf91150efac3fb9918d9"
server_key_iv: "2eb1172c80a8d90234e8baa2c54d6673"
server_key_version: "encryptionkey@0"
I20260808 22:24:47.434969 1229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:24:47.460232 1229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:24:47.463179 1229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:24:47.463353 1229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:24:47.463950 1229 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:24:47.464783 1229 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:24:47.464833 1229 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:24:47.464886 1229 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:24:47.464942 1229 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:24:47.473626 1229 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.194:35133
I20260808 22:24:47.473661 1418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.194:35133 every 8 connection(s)
I20260808 22:24:47.474455 1229 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
I20260808 22:24:47.475373 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 1229
I20260808 22:24:47.475487 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-1/wal/instance
I20260808 22:24:47.475744 9923 external_mini_cluster.cc:1428] Setting key 607d9d65d2f6e5bb3f24d0e9d1b332f3
Aug 08 22:24:47 dist-test-slave-ncp9 krb5kdc[860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227887, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.194@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.176.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.176.195@KRBTEST.COM" created.
I20260808 22:24:47.488929 986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:47.476244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.194:51753 (local address 127.9.176.254:39769)
0808 22:24:47.476381 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0808 22:24:47.476386 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:24:47.477392 (+ 1006us) server_negotiation.cc:407] Connection header received
0808 22:24:47.478716 (+ 1324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:24:47.478719 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:24:47.478771 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:24:47.478851 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:24:47.480395 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:47.480946 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:47.482601 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:47.482824 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:47.482901 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:47.485559 (+ 2658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:24:47.485563 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:24:47.485566 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:24:47.485590 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:24:47.486909 (+ 1319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:24:47.487377 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:24:47.487380 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:24:47.487382 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:24:47.487458 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:24:47.487771 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:24:47.487775 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:24:47.487777 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:24:47.487990 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:24:47.488126 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0808 22:24:47.488610 (+ 484us) server_negotiation.cc:299] Negotiation successful
0808 22:24:47.488739 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 22:24:47.489707 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:47.476702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:39769 (local address 127.9.176.194:51753)
0808 22:24:47.477239 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:47.477291 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 22:24:47.478514 (+ 1223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:24:47.479071 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:24:47.479081 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:24:47.479585 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:24:47.480243 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:24:47.480264 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:24:47.481066 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:24:47.481069 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:24:47.482487 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:24:47.482500 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:24:47.482977 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:24:47.482981 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:24:47.483205 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:47.483763 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:24:47.483785 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:24:47.485400 (+ 1615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:24:47.487021 (+ 1621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:24:47.487027 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:24:47.487042 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:24:47.487284 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:24:47.487539 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:24:47.487542 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:24:47.487544 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:24:47.487683 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:24:47.488126 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:24:47.488133 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:24:47.488363 (+ 230us) client_negotiation.cc:769] Sending connection context
0808 22:24:47.488542 (+ 179us) client_negotiation.cc:240] Negotiation successful
0808 22:24:47.488866 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":172,"threads_started":1}
I20260808 22:24:47.490857 1420 heartbeater.cc:347] Connected to a master server at 127.9.176.254:39769
I20260808 22:24:47.491127 1420 heartbeater.cc:464] Registering TS with master...
I20260808 22:24:47.491654 1420 heartbeater.cc:511] Master 127.9.176.254:39769 requested a full tablet report, sending...
I20260808 22:24:47.492767 926 ts_manager.cc:194] Registered new tserver with Master: 87604ec3fe8c43b688447d96bd4bfd3e (127.9.176.194:35133)
I20260808 22:24:47.493544 926 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.176.194@KRBTEST.COM'} at 127.9.176.194:51753
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.176.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.195.keytab.
Entry for principal kudu/127.9.176.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.195.keytab.
I20260808 22:24:47.503377 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.195:0
--local_ip_for_outbound_sockets=127.9.176.195
--webserver_interface=127.9.176.195
--webserver_port=0
--tserver_master_addrs=127.9.176.254:39769
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.195.keytab
--principal=kudu/127.9.176.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:33147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:24:47.614830 1427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:24:47.615073 1427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:24:47.615115 1427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:24:47.619072 1427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:24:47.619132 1427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:24:47.619197 1427 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.195
I20260808 22:24:47.624480 1427 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:33147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.195.keytab
--principal=kudu/127.9.176.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.176.195
--webserver_port=0
--tserver_master_addrs=127.9.176.254:39769
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1427
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.195
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:24:47.625551 1427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:24:47.626585 1427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 22:24:47.636219 1427 server_base.cc:1034] running on GCE node
W20260808 22:24:47.636188 1438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:24:47.636160 1437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:24:47.636173 1440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:24:47.636931 1427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:24:47.637570 1427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:24:47.638720 1427 hybrid_clock.cc:648] HybridClock initialized: now 1786227887638702 us; error 31 us; skew 500 ppm
Aug 08 22:24:47 dist-test-slave-ncp9 krb5kdc[860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227887, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:24:47.641496 1427 init.cc:377] Logged in from keytab as kudu/127.9.176.195@KRBTEST.COM (short username kudu)
I20260808 22:24:47.642409 1427 webserver.cc:460] Webserver started at http://127.9.176.195:36629/ using document root <none> and password file <none>
I20260808 22:24:47.642957 1427 fs_manager.cc:359] Metadata directory not provided
I20260808 22:24:47.643025 1427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:24:47.643229 1427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:24:47.645304 1427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-2/data/instance:
uuid: "a606b1ce23654e70be927e2072afa5c6"
format_stamp: "Formatted at 2026-08-08 22:24:47 on dist-test-slave-ncp9"
server_key: "11ee048a4c2fcedbcb20a9c6ecb83794"
server_key_iv: "d01037f3741c926609f3267f18ab274a"
server_key_version: "encryptionkey@0"
I20260808 22:24:47.645864 1427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-2/wal/instance:
uuid: "a606b1ce23654e70be927e2072afa5c6"
format_stamp: "Formatted at 2026-08-08 22:24:47 on dist-test-slave-ncp9"
server_key: "11ee048a4c2fcedbcb20a9c6ecb83794"
server_key_iv: "d01037f3741c926609f3267f18ab274a"
server_key_version: "encryptionkey@0"
I20260808 22:24:47.649521 1427 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260808 22:24:47.651872 1448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:24:47.653149 1427 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 22:24:47.653260 1427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-2/wal
uuid: "a606b1ce23654e70be927e2072afa5c6"
format_stamp: "Formatted at 2026-08-08 22:24:47 on dist-test-slave-ncp9"
server_key: "11ee048a4c2fcedbcb20a9c6ecb83794"
server_key_iv: "d01037f3741c926609f3267f18ab274a"
server_key_version: "encryptionkey@0"
I20260808 22:24:47.653362 1427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:24:47.670324 1427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:24:47.673050 1427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:24:47.673242 1427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:24:47.673827 1427 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:24:47.674628 1427 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:24:47.674685 1427 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:24:47.674738 1427 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:24:47.674767 1427 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:24:47.683378 1427 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.195:35701
I20260808 22:24:47.683405 1613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.195:35701 every 8 connection(s)
I20260808 22:24:47.684255 1427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
I20260808 22:24:47.685209 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 1427
I20260808 22:24:47.685313 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-2/wal/instance
I20260808 22:24:47.685562 9923 external_mini_cluster.cc:1428] Setting key 3bc42ea06605e4f1e10a83ecc6921dbe
Aug 08 22:24:47 dist-test-slave-ncp9 krb5kdc[860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227887, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.195@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
I20260808 22:24:47.697141 986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:47.686168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.195:44975 (local address 127.9.176.254:39769)
0808 22:24:47.686301 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0808 22:24:47.686306 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:24:47.687335 (+ 1029us) server_negotiation.cc:407] Connection header received
0808 22:24:47.688708 (+ 1373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:24:47.688711 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:24:47.688755 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:24:47.688819 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:24:47.689998 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:47.690540 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:47.691622 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:47.691830 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:47.691894 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:47.694294 (+ 2400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:24:47.694297 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:24:47.694299 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:24:47.694323 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:24:47.695437 (+ 1114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:24:47.695874 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:24:47.695876 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:24:47.695878 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:24:47.695943 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:24:47.696213 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:24:47.696215 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:24:47.696216 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:24:47.696381 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:24:47.696448 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0808 22:24:47.696887 (+ 439us) server_negotiation.cc:299] Negotiation successful
0808 22:24:47.697011 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260808 22:24:47.697993 1621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:47.686608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:39769 (local address 127.9.176.195:44975)
0808 22:24:47.687166 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:47.687231 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0808 22:24:47.688519 (+ 1288us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:24:47.689017 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:24:47.689025 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:24:47.689403 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:24:47.689882 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:24:47.689896 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:24:47.690632 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:24:47.690634 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:24:47.691527 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:24:47.691537 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:24:47.691934 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:24:47.691937 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:24:47.692091 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:47.692651 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:24:47.692668 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:24:47.694152 (+ 1484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:24:47.695532 (+ 1380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:24:47.695537 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:24:47.695548 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:24:47.695802 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:24:47.696006 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:24:47.696008 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:24:47.696009 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:24:47.696149 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:24:47.696450 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:24:47.696456 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:24:47.696701 (+ 245us) client_negotiation.cc:769] Sending connection context
0808 22:24:47.696859 (+ 158us) client_negotiation.cc:240] Negotiation successful
0808 22:24:47.697186 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":134,"threads_started":1}
I20260808 22:24:47.699301 1616 heartbeater.cc:347] Connected to a master server at 127.9.176.254:39769
I20260808 22:24:47.699555 1616 heartbeater.cc:464] Registering TS with master...
I20260808 22:24:47.700052 1616 heartbeater.cc:511] Master 127.9.176.254:39769 requested a full tablet report, sending...
I20260808 22:24:47.701110 925 ts_manager.cc:194] Registered new tserver with Master: a606b1ce23654e70be927e2072afa5c6 (127.9.176.195:35701)
I20260808 22:24:47.701689 925 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.176.195@KRBTEST.COM'} at 127.9.176.195:44975
I20260808 22:24:47.711355 9923 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 22:24:47.720306 986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:47.713363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38274 (local address 127.9.176.254:39769)
0808 22:24:47.713521 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0808 22:24:47.713525 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:24:47.713610 (+ 85us) server_negotiation.cc:407] Connection header received
0808 22:24:47.713714 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:24:47.713716 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:24:47.713757 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:24:47.713822 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:24:47.714465 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:47.714976 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:47.715866 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:47.716070 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:47.716137 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:47.717641 (+ 1504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:24:47.717643 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:24:47.717645 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:24:47.717669 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:24:47.718838 (+ 1169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:24:47.719237 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:24:47.719239 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:24:47.719240 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:24:47.719305 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:24:47.719583 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:24:47.719584 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:24:47.719585 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:24:47.719803 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:24:47.719900 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0808 22:24:47.720064 (+ 164us) server_negotiation.cc:299] Negotiation successful
0808 22:24:47.720158 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260808 22:24:47.723382 925 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38274:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 22:24:47.726425 925 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 22:24:47.740371 1634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:47.734481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.193:34743 (local address 127.0.0.1:33762)
0808 22:24:47.735221 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:47.735275 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0808 22:24:47.735413 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:24:47.736561 (+ 1148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:24:47.736564 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:24:47.736581 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:24:47.736673 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:24:47.736679 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:24:47.737722 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:24:47.737726 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:24:47.738968 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:24:47.738979 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:24:47.739783 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:24:47.739786 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:24:47.739939 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:47.739986 (+ 47us) client_negotiation.cc:769] Sending connection context
0808 22:24:47.740093 (+ 107us) client_negotiation.cc:240] Negotiation successful
0808 22:24:47.740212 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"mutex_wait_us":24,"spinlock_wait_cycles":13440,"thread_start_us":85,"threads_started":1}
I20260808 22:24:47.740523 1638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:47.734558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.194:35133 (local address 127.0.0.1:44122)
0808 22:24:47.735217 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:47.735275 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0808 22:24:47.735414 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:24:47.736375 (+ 961us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:24:47.736381 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:24:47.736402 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:24:47.736543 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:24:47.736550 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:24:47.738093 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:24:47.738098 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:24:47.739045 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:24:47.739064 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:24:47.740093 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:24:47.740096 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:24:47.740216 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:47.740236 (+ 20us) client_negotiation.cc:769] Sending connection context
0808 22:24:47.740307 (+ 71us) client_negotiation.cc:240] Negotiation successful
0808 22:24:47.740362 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"mutex_wait_us":50,"spinlock_wait_cycles":10496,"thread_start_us":112,"threads_started":1}
I20260808 22:24:47.740619 1639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:47.735497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44122 (local address 127.9.176.194:35133)
0808 22:24:47.735915 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0808 22:24:47.735936 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0808 22:24:47.735960 (+ 24us) server_negotiation.cc:407] Connection header received
0808 22:24:47.736040 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:24:47.736046 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:24:47.736237 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:24:47.736370 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:24:47.736709 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:47.737947 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:47.739166 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:47.739975 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:47.740078 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:47.740216 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0808 22:24:47.740347 (+ 131us) server_negotiation.cc:299] Negotiation successful
0808 22:24:47.740447 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":165,"threads_started":1}
I20260808 22:24:47.741142 1637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:47.734613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33762 (local address 127.9.176.193:34743)
0808 22:24:47.736170 (+ 1557us) server_negotiation.cc:206] Beginning negotiation
0808 22:24:47.736185 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 22:24:47.736204 (+ 19us) server_negotiation.cc:407] Connection header received
0808 22:24:47.736269 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:24:47.736273 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:24:47.736413 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:24:47.736575 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:24:47.736772 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:47.737596 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:47.739084 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:47.739679 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:47.739780 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:47.739937 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0808 22:24:47.740850 (+ 913us) server_negotiation.cc:299] Negotiation successful
0808 22:24:47.740984 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1444,"thread_start_us":66,"threads_started":1}
I20260808 22:24:47.743907 1117 tablet_service.cc:1467] Processing CreateTablet for tablet e8c85cd9c93247edb3b4a8350db7a5ec (DEFAULT_TABLE table=test-table [id=abfd36887513494aa55376a91bfc02ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:24:47.743907 1326 tablet_service.cc:1467] Processing CreateTablet for tablet e8c85cd9c93247edb3b4a8350db7a5ec (DEFAULT_TABLE table=test-table [id=abfd36887513494aa55376a91bfc02ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:24:47.744773 1635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:47.734868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.195:35701 (local address 127.0.0.1:57758)
0808 22:24:47.735930 (+ 1062us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:47.735943 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 22:24:47.736025 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:24:47.736673 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:24:47.736676 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:24:47.736694 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:24:47.736786 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:24:47.736792 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:24:47.738198 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:24:47.738201 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:24:47.740175 (+ 1974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:24:47.740186 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:24:47.741398 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:24:47.741401 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:24:47.741501 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:47.741536 (+ 35us) client_negotiation.cc:769] Sending connection context
0808 22:24:47.744575 (+ 3039us) client_negotiation.cc:240] Negotiation successful
0808 22:24:47.744669 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1008,"mutex_wait_us":4,"thread_start_us":77,"threads_started":1}
I20260808 22:24:47.745134 1117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8c85cd9c93247edb3b4a8350db7a5ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:24:47.745137 1326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8c85cd9c93247edb3b4a8350db7a5ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:24:47.745220 1640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:47.735681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57758 (local address 127.9.176.195:35701)
0808 22:24:47.736039 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0808 22:24:47.736051 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 22:24:47.736074 (+ 23us) server_negotiation.cc:407] Connection header received
0808 22:24:47.736150 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:24:47.736155 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:24:47.736347 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:24:47.736496 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:24:47.736937 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:47.738069 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:47.740308 (+ 2239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:47.741109 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:47.741204 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:47.741336 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0808 22:24:47.744957 (+ 3621us) server_negotiation.cc:299] Negotiation successful
0808 22:24:47.745074 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":149,"threads_started":1}
I20260808 22:24:47.748116 1513 tablet_service.cc:1467] Processing CreateTablet for tablet e8c85cd9c93247edb3b4a8350db7a5ec (DEFAULT_TABLE table=test-table [id=abfd36887513494aa55376a91bfc02ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:24:47.749269 1513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8c85cd9c93247edb3b4a8350db7a5ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:24:47.752727 1641 tablet_bootstrap.cc:492] T e8c85cd9c93247edb3b4a8350db7a5ec P 87604ec3fe8c43b688447d96bd4bfd3e: Bootstrap starting.
I20260808 22:24:47.754608 1642 tablet_bootstrap.cc:492] T e8c85cd9c93247edb3b4a8350db7a5ec P 75595c6cc3d54995ade109df7bbe8820: Bootstrap starting.
I20260808 22:24:47.755753 1641 tablet_bootstrap.cc:654] T e8c85cd9c93247edb3b4a8350db7a5ec P 87604ec3fe8c43b688447d96bd4bfd3e: Neither blocks nor log segments found. Creating new log.
I20260808 22:24:47.756523 1643 tablet_bootstrap.cc:492] T e8c85cd9c93247edb3b4a8350db7a5ec P a606b1ce23654e70be927e2072afa5c6: Bootstrap starting.
I20260808 22:24:47.756863 1642 tablet_bootstrap.cc:654] T e8c85cd9c93247edb3b4a8350db7a5ec P 75595c6cc3d54995ade109df7bbe8820: Neither blocks nor log segments found. Creating new log.
I20260808 22:24:47.756928 1641 log.cc:824] T e8c85cd9c93247edb3b4a8350db7a5ec P 87604ec3fe8c43b688447d96bd4bfd3e: Log is configured to *not* fsync() on all Append() calls
I20260808 22:24:47.758248 1642 log.cc:824] T e8c85cd9c93247edb3b4a8350db7a5ec P 75595c6cc3d54995ade109df7bbe8820: Log is configured to *not* fsync() on all Append() calls
I20260808 22:24:47.758745 1643 tablet_bootstrap.cc:654] T e8c85cd9c93247edb3b4a8350db7a5ec P a606b1ce23654e70be927e2072afa5c6: Neither blocks nor log segments found. Creating new log.
I20260808 22:24:47.759303 1641 tablet_bootstrap.cc:492] T e8c85cd9c93247edb3b4a8350db7a5ec P 87604ec3fe8c43b688447d96bd4bfd3e: No bootstrap required, opened a new log
I20260808 22:24:47.759516 1641 ts_tablet_manager.cc:1397] T e8c85cd9c93247edb3b4a8350db7a5ec P 87604ec3fe8c43b688447d96bd4bfd3e: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260808 22:24:47.759934 1642 tablet_bootstrap.cc:492] T e8c85cd9c93247edb3b4a8350db7a5ec P 75595c6cc3d54995ade109df7bbe8820: No bootstrap required, opened a new log
I20260808 22:24:47.760120 1642 ts_tablet_manager.cc:1397] T e8c85cd9c93247edb3b4a8350db7a5ec P 75595c6cc3d54995ade109df7bbe8820: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 22:24:47.760210 1643 log.cc:824] T e8c85cd9c93247edb3b4a8350db7a5ec P a606b1ce23654e70be927e2072afa5c6: Log is configured to *not* fsync() on all Append() calls
I20260808 22:24:47.762163 1643 tablet_bootstrap.cc:492] T e8c85cd9c93247edb3b4a8350db7a5ec P a606b1ce23654e70be927e2072afa5c6: No bootstrap required, opened a new log
I20260808 22:24:47.762379 1643 ts_tablet_manager.cc:1397] T e8c85cd9c93247edb3b4a8350db7a5ec P a606b1ce23654e70be927e2072afa5c6: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 22:24:47.763303 1642 raft_consensus.cc:348] T e8c85cd9c93247edb3b4a8350db7a5ec P 75595c6cc3d54995ade109df7bbe8820 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75595c6cc3d54995ade109df7bbe8820" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34743 } } peers { permanent_uuid: "87604ec3fe8c43b688447d96bd4bfd3e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 35133 } } peers { permanent_uuid: "a606b1ce23654e70be927e2072afa5c6" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 35701 } }
I20260808 22:24:47.763314 1641 raft_consensus.cc:348] T e8c85cd9c93247edb3b4a8350db7a5ec P 87604ec3fe8c43b688447d96bd4bfd3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75595c6cc3d54995ade109df7bbe8820" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34743 } } peers { permanent_uuid: "87604ec3fe8c43b688447d96bd4bfd3e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 35133 } } peers { permanent_uuid: "a606b1ce23654e70be927e2072afa5c6" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 35701 } }
I20260808 22:24:47.763535 1642 raft_consensus.cc:374] T e8c85cd9c93247edb3b4a8350db7a5ec P 75595c6cc3d54995ade109df7bbe8820 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:24:47.763571 1641 raft_consensus.cc:374] T e8c85cd9c93247edb3b4a8350db7a5ec P 87604ec3fe8c43b688447d96bd4bfd3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:24:47.763588 1642 raft_consensus.cc:729] T e8c85cd9c93247edb3b4a8350db7a5ec P 75595c6cc3d54995ade109df7bbe8820 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75595c6cc3d54995ade109df7bbe8820, State: Initialized, Role: FOLLOWER
I20260808 22:24:47.763628 1641 raft_consensus.cc:729] T e8c85cd9c93247edb3b4a8350db7a5ec P 87604ec3fe8c43b688447d96bd4bfd3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87604ec3fe8c43b688447d96bd4bfd3e, State: Initialized, Role: FOLLOWER
I20260808 22:24:47.763994 1642 consensus_queue.cc:260] T e8c85cd9c93247edb3b4a8350db7a5ec P 75595c6cc3d54995ade109df7bbe8820 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75595c6cc3d54995ade109df7bbe8820" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34743 } } peers { permanent_uuid: "87604ec3fe8c43b688447d96bd4bfd3e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 35133 } } peers { permanent_uuid: "a606b1ce23654e70be927e2072afa5c6" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 35701 } }
I20260808 22:24:47.764096 1641 consensus_queue.cc:260] T e8c85cd9c93247edb3b4a8350db7a5ec P 87604ec3fe8c43b688447d96bd4bfd3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75595c6cc3d54995ade109df7bbe8820" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34743 } } peers { permanent_uuid: "87604ec3fe8c43b688447d96bd4bfd3e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 35133 } } peers { permanent_uuid: "a606b1ce23654e70be927e2072afa5c6" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 35701 } }
I20260808 22:24:47.764624 1222 heartbeater.cc:503] Master 127.9.176.254:39769 was elected leader, sending a full tablet report...
I20260808 22:24:47.765062 1642 ts_tablet_manager.cc:1428] T e8c85cd9c93247edb3b4a8350db7a5ec P 75595c6cc3d54995ade109df7bbe8820: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260808 22:24:47.765341 1420 heartbeater.cc:503] Master 127.9.176.254:39769 was elected leader, sending a full tablet report...
I20260808 22:24:47.766187 1641 ts_tablet_manager.cc:1428] T e8c85cd9c93247edb3b4a8350db7a5ec P 87604ec3fe8c43b688447d96bd4bfd3e: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260808 22:24:47.766542 1643 raft_consensus.cc:348] T e8c85cd9c93247edb3b4a8350db7a5ec P a606b1ce23654e70be927e2072afa5c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75595c6cc3d54995ade109df7bbe8820" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34743 } } peers { permanent_uuid: "87604ec3fe8c43b688447d96bd4bfd3e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 35133 } } peers { permanent_uuid: "a606b1ce23654e70be927e2072afa5c6" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 35701 } }
I20260808 22:24:47.766739 1643 raft_consensus.cc:374] T e8c85cd9c93247edb3b4a8350db7a5ec P a606b1ce23654e70be927e2072afa5c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:24:47.766782 1643 raft_consensus.cc:729] T e8c85cd9c93247edb3b4a8350db7a5ec P a606b1ce23654e70be927e2072afa5c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a606b1ce23654e70be927e2072afa5c6, State: Initialized, Role: FOLLOWER
I20260808 22:24:47.767143 1643 consensus_queue.cc:260] T e8c85cd9c93247edb3b4a8350db7a5ec P a606b1ce23654e70be927e2072afa5c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75595c6cc3d54995ade109df7bbe8820" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34743 } } peers { permanent_uuid: "87604ec3fe8c43b688447d96bd4bfd3e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 35133 } } peers { permanent_uuid: "a606b1ce23654e70be927e2072afa5c6" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 35701 } }
I20260808 22:24:47.767731 1616 heartbeater.cc:503] Master 127.9.176.254:39769 was elected leader, sending a full tablet report...
I20260808 22:24:47.768019 1643 ts_tablet_manager.cc:1428] T e8c85cd9c93247edb3b4a8350db7a5ec P a606b1ce23654e70be927e2072afa5c6: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260808 22:24:47.770308 1647 raft_consensus.cc:482] T e8c85cd9c93247edb3b4a8350db7a5ec P 75595c6cc3d54995ade109df7bbe8820 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 22:24:47.770474 1647 raft_consensus.cc:504] T e8c85cd9c93247edb3b4a8350db7a5ec P 75595c6cc3d54995ade109df7bbe8820 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75595c6cc3d54995ade109df7bbe8820" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34743 } } peers { permanent_uuid: "87604ec3fe8c43b688447d96bd4bfd3e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 35133 } } peers { permanent_uuid: "a606b1ce23654e70be927e2072afa5c6" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 35701 } }
I20260808 22:24:47.771642 1647 leader_election.cc:288] T e8c85cd9c93247edb3b4a8350db7a5ec P 75595c6cc3d54995ade109df7bbe8820 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87604ec3fe8c43b688447d96bd4bfd3e (127.9.176.194:35133), a606b1ce23654e70be927e2072afa5c6 (127.9.176.195:35701)
I20260808 22:24:47.775913 1225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:47.771846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.194:35133 (local address 127.9.176.193:48805)
0808 22:24:47.772100 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:47.772116 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 22:24:47.772199 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:24:47.772719 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:24:47.772722 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:24:47.772744 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:24:47.772872 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:24:47.772879 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:24:47.773801 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:24:47.773804 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:24:47.774686 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:24:47.774697 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:24:47.775476 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:24:47.775479 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:24:47.775593 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:47.775620 (+ 27us) client_negotiation.cc:769] Sending connection context
0808 22:24:47.775706 (+ 86us) client_negotiation.cc:240] Negotiation successful
0808 22:24:47.775782 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176}
I20260808 22:24:47.775959 1639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:47.772353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.193:48805 (local address 127.9.176.194:35133)
0808 22:24:47.772477 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0808 22:24:47.772483 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 22:24:47.772498 (+ 15us) server_negotiation.cc:407] Connection header received
0808 22:24:47.772549 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:24:47.772552 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:24:47.772600 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:24:47.772690 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:24:47.773031 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:47.773688 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:47.774822 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:47.775364 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:47.775463 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:47.775511 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0808 22:24:47.775735 (+ 224us) server_negotiation.cc:299] Negotiation successful
0808 22:24:47.775810 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 22:24:47.776412 1640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:47.771930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.193:40775 (local address 127.9.176.195:35701)
0808 22:24:47.772041 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0808 22:24:47.772045 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:24:47.772921 (+ 876us) server_negotiation.cc:407] Connection header received
0808 22:24:47.773060 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:24:47.773062 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:24:47.773098 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:24:47.773148 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:24:47.773437 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:47.773991 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:47.775200 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:47.775671 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:47.775777 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:47.775820 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0808 22:24:47.776227 (+ 407us) server_negotiation.cc:299] Negotiation successful
0808 22:24:47.776305 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260808 22:24:47.776410 1650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:47.771869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.195:35701 (local address 127.9.176.193:40775)
0808 22:24:47.772807 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:47.772821 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 22:24:47.772952 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:24:47.773178 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:24:47.773181 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:24:47.773200 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:24:47.773340 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:24:47.773348 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:24:47.774118 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:24:47.774120 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:24:47.775069 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:24:47.775082 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:24:47.775782 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:24:47.775787 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:24:47.775902 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:47.775930 (+ 28us) client_negotiation.cc:769] Sending connection context
0808 22:24:47.776189 (+ 259us) client_negotiation.cc:240] Negotiation successful
0808 22:24:47.776266 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":868,"mutex_wait_us":1,"thread_start_us":108,"threads_started":1}
I20260808 22:24:47.776566 1354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8c85cd9c93247edb3b4a8350db7a5ec" candidate_uuid: "75595c6cc3d54995ade109df7bbe8820" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87604ec3fe8c43b688447d96bd4bfd3e" is_pre_election: true
I20260808 22:24:47.776852 1354 raft_consensus.cc:2454] T e8c85cd9c93247edb3b4a8350db7a5ec P 87604ec3fe8c43b688447d96bd4bfd3e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75595c6cc3d54995ade109df7bbe8820 in term 0.
I20260808 22:24:47.777031 1543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8c85cd9c93247edb3b4a8350db7a5ec" candidate_uuid: "75595c6cc3d54995ade109df7bbe8820" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a606b1ce23654e70be927e2072afa5c6" is_pre_election: true
I20260808 22:24:47.777315 1543 raft_consensus.cc:2454] T e8c85cd9c93247edb3b4a8350db7a5ec P a606b1ce23654e70be927e2072afa5c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75595c6cc3d54995ade109df7bbe8820 in term 0.
I20260808 22:24:47.777412 1050 leader_election.cc:302] T e8c85cd9c93247edb3b4a8350db7a5ec P 75595c6cc3d54995ade109df7bbe8820 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75595c6cc3d54995ade109df7bbe8820, 87604ec3fe8c43b688447d96bd4bfd3e; no voters:
I20260808 22:24:47.777674 1647 raft_consensus.cc:2781] T e8c85cd9c93247edb3b4a8350db7a5ec P 75595c6cc3d54995ade109df7bbe8820 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 22:24:47.777765 1647 raft_consensus.cc:482] T e8c85cd9c93247edb3b4a8350db7a5ec P 75595c6cc3d54995ade109df7bbe8820 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 22:24:47.777804 1647 raft_consensus.cc:3037] T e8c85cd9c93247edb3b4a8350db7a5ec P 75595c6cc3d54995ade109df7bbe8820 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:24:47.778865 1647 raft_consensus.cc:504] T e8c85cd9c93247edb3b4a8350db7a5ec P 75595c6cc3d54995ade109df7bbe8820 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75595c6cc3d54995ade109df7bbe8820" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34743 } } peers { permanent_uuid: "87604ec3fe8c43b688447d96bd4bfd3e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 35133 } } peers { permanent_uuid: "a606b1ce23654e70be927e2072afa5c6" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 35701 } }
I20260808 22:24:47.779241 1647 leader_election.cc:288] T e8c85cd9c93247edb3b4a8350db7a5ec P 75595c6cc3d54995ade109df7bbe8820 [CANDIDATE]: Term 1 election: Requested vote from peers 87604ec3fe8c43b688447d96bd4bfd3e (127.9.176.194:35133), a606b1ce23654e70be927e2072afa5c6 (127.9.176.195:35701)
I20260808 22:24:47.779574 1543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8c85cd9c93247edb3b4a8350db7a5ec" candidate_uuid: "75595c6cc3d54995ade109df7bbe8820" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a606b1ce23654e70be927e2072afa5c6"
I20260808 22:24:47.779573 1354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8c85cd9c93247edb3b4a8350db7a5ec" candidate_uuid: "75595c6cc3d54995ade109df7bbe8820" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87604ec3fe8c43b688447d96bd4bfd3e"
I20260808 22:24:47.779700 1354 raft_consensus.cc:3037] T e8c85cd9c93247edb3b4a8350db7a5ec P 87604ec3fe8c43b688447d96bd4bfd3e [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:24:47.779700 1543 raft_consensus.cc:3037] T e8c85cd9c93247edb3b4a8350db7a5ec P a606b1ce23654e70be927e2072afa5c6 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:24:47.781018 1354 raft_consensus.cc:2454] T e8c85cd9c93247edb3b4a8350db7a5ec P 87604ec3fe8c43b688447d96bd4bfd3e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75595c6cc3d54995ade109df7bbe8820 in term 1.
I20260808 22:24:47.781018 1543 raft_consensus.cc:2454] T e8c85cd9c93247edb3b4a8350db7a5ec P a606b1ce23654e70be927e2072afa5c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75595c6cc3d54995ade109df7bbe8820 in term 1.
I20260808 22:24:47.781315 1050 leader_election.cc:302] T e8c85cd9c93247edb3b4a8350db7a5ec P 75595c6cc3d54995ade109df7bbe8820 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75595c6cc3d54995ade109df7bbe8820, 87604ec3fe8c43b688447d96bd4bfd3e; no voters:
I20260808 22:24:47.781515 1647 raft_consensus.cc:2781] T e8c85cd9c93247edb3b4a8350db7a5ec P 75595c6cc3d54995ade109df7bbe8820 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:24:47.781734 1647 raft_consensus.cc:686] T e8c85cd9c93247edb3b4a8350db7a5ec P 75595c6cc3d54995ade109df7bbe8820 [term 1 LEADER]: Becoming Leader. State: Replica: 75595c6cc3d54995ade109df7bbe8820, State: Running, Role: LEADER
I20260808 22:24:47.782035 1647 consensus_queue.cc:237] T e8c85cd9c93247edb3b4a8350db7a5ec P 75595c6cc3d54995ade109df7bbe8820 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75595c6cc3d54995ade109df7bbe8820" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34743 } } peers { permanent_uuid: "87604ec3fe8c43b688447d96bd4bfd3e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 35133 } } peers { permanent_uuid: "a606b1ce23654e70be927e2072afa5c6" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 35701 } }
I20260808 22:24:47.785265 923 catalog_manager.cc:5495] T e8c85cd9c93247edb3b4a8350db7a5ec P 75595c6cc3d54995ade109df7bbe8820 reported cstate change: term changed from 0 to 1, leader changed from <none> to 75595c6cc3d54995ade109df7bbe8820 (127.9.176.193). New cstate: current_term: 1 leader_uuid: "75595c6cc3d54995ade109df7bbe8820" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75595c6cc3d54995ade109df7bbe8820" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34743 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "87604ec3fe8c43b688447d96bd4bfd3e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 35133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a606b1ce23654e70be927e2072afa5c6" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 35701 } health_report { overall_health: UNKNOWN } } }
I20260808 22:24:47.802731 1637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:47.799684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33778 (local address 127.9.176.193:34743)
0808 22:24:47.799797 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0808 22:24:47.799801 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:24:47.799832 (+ 31us) server_negotiation.cc:407] Connection header received
0808 22:24:47.799898 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:24:47.799900 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:24:47.799947 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:24:47.800020 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:24:47.800211 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:47.800704 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:47.801528 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:24:47.801741 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:24:47.801808 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:24:47.802028 (+ 220us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:24:47.802444 (+ 416us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:24:47.802529 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0808 22:24:47.802556 (+ 27us) server_negotiation.cc:299] Negotiation successful
0808 22:24:47.802605 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 22:24:47.808281 1543 raft_consensus.cc:1261] T e8c85cd9c93247edb3b4a8350db7a5ec P a606b1ce23654e70be927e2072afa5c6 [term 1 FOLLOWER]: Refusing update from remote peer 75595c6cc3d54995ade109df7bbe8820: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:24:47.808281 1354 raft_consensus.cc:1261] T e8c85cd9c93247edb3b4a8350db7a5ec P 87604ec3fe8c43b688447d96bd4bfd3e [term 1 FOLLOWER]: Refusing update from remote peer 75595c6cc3d54995ade109df7bbe8820: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:24:47.808940 1653 consensus_queue.cc:1035] T e8c85cd9c93247edb3b4a8350db7a5ec P 75595c6cc3d54995ade109df7bbe8820 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87604ec3fe8c43b688447d96bd4bfd3e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 35133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:24:47.809118 1647 consensus_queue.cc:1035] T e8c85cd9c93247edb3b4a8350db7a5ec P 75595c6cc3d54995ade109df7bbe8820 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a606b1ce23654e70be927e2072afa5c6" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 35701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:24:47.817600 1660 mvcc.cc:205] Tried to move back new op lower bound from 7316389428455723008 to 7316389428358311936. Current Snapshot: MvccSnapshot[applied={T|T < 7316389428455723008}]
I20260808 22:24:47.817807 1659 mvcc.cc:205] Tried to move back new op lower bound from 7316389428455723008 to 7316389428358311936. Current Snapshot: MvccSnapshot[applied={T|T < 7316389428455723008}]
I20260808 22:24:47.825518 923 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38274:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:24:47.825732 923 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38274:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:24:47.828302 923 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d2f7a8bd3d904d54b493540ee9062e2b: Sending DeleteTablet for 3 replicas of tablet e8c85cd9c93247edb3b4a8350db7a5ec
I20260808 22:24:47.829066 1117 tablet_service.cc:1514] Processing DeleteTablet for tablet e8c85cd9c93247edb3b4a8350db7a5ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:24:47 UTC) from {username='kudu', principal='kudu/127.9.176.254@KRBTEST.COM'} at 127.0.0.1:33762
I20260808 22:24:47.829164 1326 tablet_service.cc:1514] Processing DeleteTablet for tablet e8c85cd9c93247edb3b4a8350db7a5ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:24:47 UTC) from {username='kudu', principal='kudu/127.9.176.254@KRBTEST.COM'} at 127.0.0.1:44122
I20260808 22:24:47.829236 1513 tablet_service.cc:1514] Processing DeleteTablet for tablet e8c85cd9c93247edb3b4a8350db7a5ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:24:47 UTC) from {username='kudu', principal='kudu/127.9.176.254@KRBTEST.COM'} at 127.0.0.1:57758
I20260808 22:24:47.829643 1674 tablet_replica.cc:330] T e8c85cd9c93247edb3b4a8350db7a5ec P a606b1ce23654e70be927e2072afa5c6: stopping tablet replica
I20260808 22:24:47.829685 1675 tablet_replica.cc:330] T e8c85cd9c93247edb3b4a8350db7a5ec P 87604ec3fe8c43b688447d96bd4bfd3e: stopping tablet replica
I20260808 22:24:47.829965 1674 raft_consensus.cc:2229] T e8c85cd9c93247edb3b4a8350db7a5ec P a606b1ce23654e70be927e2072afa5c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:24:47.829985 1675 raft_consensus.cc:2229] T e8c85cd9c93247edb3b4a8350db7a5ec P 87604ec3fe8c43b688447d96bd4bfd3e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:24:47.830096 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 1024
I20260808 22:24:47.830291 1674 raft_consensus.cc:2258] T e8c85cd9c93247edb3b4a8350db7a5ec P a606b1ce23654e70be927e2072afa5c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:24:47.830289 1675 raft_consensus.cc:2258] T e8c85cd9c93247edb3b4a8350db7a5ec P 87604ec3fe8c43b688447d96bd4bfd3e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:24:47.838917 1674 ts_tablet_manager.cc:1905] T e8c85cd9c93247edb3b4a8350db7a5ec P a606b1ce23654e70be927e2072afa5c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:24:47.840449 1675 ts_tablet_manager.cc:1905] T e8c85cd9c93247edb3b4a8350db7a5ec P 87604ec3fe8c43b688447d96bd4bfd3e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:24:47.842227 1674 ts_tablet_manager.cc:1918] T e8c85cd9c93247edb3b4a8350db7a5ec P a606b1ce23654e70be927e2072afa5c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 22:24:47.842352 1674 log.cc:1196] T e8c85cd9c93247edb3b4a8350db7a5ec P a606b1ce23654e70be927e2072afa5c6: Deleting WAL directory at /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-2/wal/wals/e8c85cd9c93247edb3b4a8350db7a5ec
I20260808 22:24:47.842782 1674 ts_tablet_manager.cc:1939] T e8c85cd9c93247edb3b4a8350db7a5ec P a606b1ce23654e70be927e2072afa5c6: Deleting consensus metadata
I20260808 22:24:47.843329 1675 ts_tablet_manager.cc:1918] T e8c85cd9c93247edb3b4a8350db7a5ec P 87604ec3fe8c43b688447d96bd4bfd3e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 22:24:47.843439 1675 log.cc:1196] T e8c85cd9c93247edb3b4a8350db7a5ec P 87604ec3fe8c43b688447d96bd4bfd3e: Deleting WAL directory at /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-1/wal/wals/e8c85cd9c93247edb3b4a8350db7a5ec
I20260808 22:24:47.843786 1675 ts_tablet_manager.cc:1939] T e8c85cd9c93247edb3b4a8350db7a5ec P 87604ec3fe8c43b688447d96bd4bfd3e: Deleting consensus metadata
I20260808 22:24:47.844280 903 catalog_manager.cc:4841] TS a606b1ce23654e70be927e2072afa5c6 (127.9.176.195:35701): tablet e8c85cd9c93247edb3b4a8350db7a5ec (table test-table [id=abfd36887513494aa55376a91bfc02ef]) successfully deleted
I20260808 22:24:47.845002 906 catalog_manager.cc:4841] TS 87604ec3fe8c43b688447d96bd4bfd3e (127.9.176.194:35133): tablet e8c85cd9c93247edb3b4a8350db7a5ec (table test-table [id=abfd36887513494aa55376a91bfc02ef]) successfully deleted
I20260808 22:24:47.845558 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 1229
W20260808 22:24:47.846112 905 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.176.193:34743 (error 108)
I20260808 22:24:47.848049 1635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:24:47.847728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.193:34743 (local address 127.0.0.1:33782)
0808 22:24:47.847900 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0808 22:24:47.847964 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.193:34743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
W20260808 22:24:47.848242 905 catalog_manager.cc:4584] TS 75595c6cc3d54995ade109df7bbe8820 (127.9.176.193:34743): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e8c85cd9c93247edb3b4a8350db7a5ec: Network error: Client connection negotiation failed: client connection to 127.9.176.193:34743: connect: Connection refused (error 111)
I20260808 22:24:47.853230 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 1427
I20260808 22:24:47.863951 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 878
I20260808 22:25:07.876545 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:39769
--webserver_interface=127.9.176.254
--webserver_port=39755
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab
--principal=kudu/127.9.176.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:33147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.176.254:39769
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:25:07.984620 1679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:25:07.984858 1679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:25:07.984932 1679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:25:07.988781 1679 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260808 22:25:07.988826 1679 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260808 22:25:07.988845 1679 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260808 22:25:07.988863 1679 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:25:07.988885 1679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:25:07.988929 1679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:25:07.988957 1679 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:25:07.988984 1679 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:25:07.993990 1679 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:33147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.9.176.254:39769
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab
--principal=kudu/127.9.176.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:39769
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.176.254
--webserver_port=39755
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1679
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:25:07.994915 1679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:25:07.995857 1679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:25:08.009639 1687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:25:08.009740 1679 server_base.cc:1034] running on GCE node
W20260808 22:25:08.009745 1688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:25:08.009650 1691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:25:08.010547 1679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:25:08.011737 1679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:25:08.012926 1679 hybrid_clock.cc:648] HybridClock initialized: now 1786227908012901 us; error 35 us; skew 500 ppm
Aug 08 22:25:08 dist-test-slave-ncp9 krb5kdc[860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227908, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:25:08.015878 1679 init.cc:377] Logged in from keytab as kudu/127.9.176.254@KRBTEST.COM (short username kudu)
I20260808 22:25:08.016834 1679 webserver.cc:460] Webserver started at http://127.9.176.254:39755/ using document root <none> and password file <none>
I20260808 22:25:08.017426 1679 fs_manager.cc:359] Metadata directory not provided
I20260808 22:25:08.017542 1679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:25:08.021504 1679 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 22:25:08.023607 1701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:25:08.024868 1679 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 22:25:08.025038 1679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/master-0/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/master-0/wal
uuid: "d2f7a8bd3d904d54b493540ee9062e2b"
format_stamp: "Formatted at 2026-08-08 22:24:46 on dist-test-slave-ncp9"
server_key: "bf076ba4a10ac6e06aecea6b2951bf93"
server_key_iv: "416756c26c90b7cc146d954b67ace62e"
server_key_version: "encryptionkey@0"
I20260808 22:25:08.025662 1679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:25:08.039083 1679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:25:08.042157 1679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:25:08.042361 1679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:25:08.049240 1679 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.254:39769
I20260808 22:25:08.049521 1774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.254:39769 every 8 connection(s)
I20260808 22:25:08.050369 1679 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
I20260808 22:25:08.053858 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 1679
I20260808 22:25:08.054405 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.193:34743
--local_ip_for_outbound_sockets=127.9.176.193
--tserver_master_addrs=127.9.176.254:39769
--webserver_port=43873
--webserver_interface=127.9.176.193
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.193.keytab
--principal=kudu/127.9.176.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:33147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 22:25:08.055308 1775 sys_catalog.cc:263] Verifying existing consensus state
I20260808 22:25:08.057314 1775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2f7a8bd3d904d54b493540ee9062e2b: Bootstrap starting.
I20260808 22:25:08.074985 1775 log.cc:824] T 00000000000000000000000000000000 P d2f7a8bd3d904d54b493540ee9062e2b: Log is configured to *not* fsync() on all Append() calls
I20260808 22:25:08.083722 1775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2f7a8bd3d904d54b493540ee9062e2b: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 22:25:08.084066 1775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2f7a8bd3d904d54b493540ee9062e2b: Bootstrap complete.
I20260808 22:25:08.088018 1775 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2f7a8bd3d904d54b493540ee9062e2b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2f7a8bd3d904d54b493540ee9062e2b" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 39769 } }
I20260808 22:25:08.088407 1775 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2f7a8bd3d904d54b493540ee9062e2b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2f7a8bd3d904d54b493540ee9062e2b, State: Initialized, Role: FOLLOWER
I20260808 22:25:08.088907 1775 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2f7a8bd3d904d54b493540ee9062e2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2f7a8bd3d904d54b493540ee9062e2b" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 39769 } }
I20260808 22:25:08.089040 1775 raft_consensus.cc:388] T 00000000000000000000000000000000 P d2f7a8bd3d904d54b493540ee9062e2b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:25:08.089113 1775 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2f7a8bd3d904d54b493540ee9062e2b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:25:08.089210 1775 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2f7a8bd3d904d54b493540ee9062e2b [term 1 FOLLOWER]: Advancing to term 2
I20260808 22:25:08.091059 1775 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2f7a8bd3d904d54b493540ee9062e2b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2f7a8bd3d904d54b493540ee9062e2b" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 39769 } }
I20260808 22:25:08.091380 1775 leader_election.cc:302] T 00000000000000000000000000000000 P d2f7a8bd3d904d54b493540ee9062e2b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2f7a8bd3d904d54b493540ee9062e2b; no voters:
I20260808 22:25:08.091681 1775 leader_election.cc:288] T 00000000000000000000000000000000 P d2f7a8bd3d904d54b493540ee9062e2b [CANDIDATE]: Term 2 election: Requested vote from peers
I20260808 22:25:08.091763 1781 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2f7a8bd3d904d54b493540ee9062e2b [term 2 FOLLOWER]: Leader election won for term 2
I20260808 22:25:08.091979 1781 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2f7a8bd3d904d54b493540ee9062e2b [term 2 LEADER]: Becoming Leader. State: Replica: d2f7a8bd3d904d54b493540ee9062e2b, State: Running, Role: LEADER
I20260808 22:25:08.092341 1781 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2f7a8bd3d904d54b493540ee9062e2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2f7a8bd3d904d54b493540ee9062e2b" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 39769 } }
I20260808 22:25:08.092929 1775 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2f7a8bd3d904d54b493540ee9062e2b [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:25:08.093415 1781 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2f7a8bd3d904d54b493540ee9062e2b [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2f7a8bd3d904d54b493540ee9062e2b. Latest consensus state: current_term: 2 leader_uuid: "d2f7a8bd3d904d54b493540ee9062e2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2f7a8bd3d904d54b493540ee9062e2b" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 39769 } } }
I20260808 22:25:08.093521 1781 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2f7a8bd3d904d54b493540ee9062e2b [sys.catalog]: This master's current role is: LEADER
I20260808 22:25:08.093705 1783 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2f7a8bd3d904d54b493540ee9062e2b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d2f7a8bd3d904d54b493540ee9062e2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2f7a8bd3d904d54b493540ee9062e2b" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 39769 } } }
I20260808 22:25:08.093823 1783 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2f7a8bd3d904d54b493540ee9062e2b [sys.catalog]: This master's current role is: LEADER
I20260808 22:25:08.094070 1787 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:25:08.098754 1787 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:25:08.099520 1787 catalog_manager.cc:1259] Loaded cluster ID: df74ca17b0754fd1913c4823f612329f
I20260808 22:25:08.099568 1787 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:25:08.101460 1787 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:25:08.102033 1787 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d2f7a8bd3d904d54b493540ee9062e2b: Loaded TSK: 0
I20260808 22:25:08.116552 1787 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d2f7a8bd3d904d54b493540ee9062e2b: Generated new TSK 1
I20260808 22:25:08.116806 1787 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 22:25:08.182996 1779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:25:08.183243 1779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:25:08.183300 1779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:25:08.187181 1779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:25:08.187234 1779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:25:08.187312 1779 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.193
I20260808 22:25:08.192384 1779 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:33147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.193.keytab
--principal=kudu/127.9.176.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.193:34743
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.176.193
--webserver_port=43873
--tserver_master_addrs=127.9.176.254:39769
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1779
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.193
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:25:08.193483 1779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:25:08.194607 1779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:25:08.203984 1816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:25:08.204047 1813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:25:08.204111 1814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:25:08.204306 1779 server_base.cc:1034] running on GCE node
I20260808 22:25:08.204793 1779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:25:08.205456 1779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:25:08.206630 1779 hybrid_clock.cc:648] HybridClock initialized: now 1786227908206610 us; error 38 us; skew 500 ppm
Aug 08 22:25:08 dist-test-slave-ncp9 krb5kdc[860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227908, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:25:08.209273 1779 init.cc:377] Logged in from keytab as kudu/127.9.176.193@KRBTEST.COM (short username kudu)
I20260808 22:25:08.210081 1779 webserver.cc:460] Webserver started at http://127.9.176.193:43873/ using document root <none> and password file <none>
I20260808 22:25:08.210575 1779 fs_manager.cc:359] Metadata directory not provided
I20260808 22:25:08.210644 1779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:25:08.214720 1779 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260808 22:25:08.216848 1826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:25:08.218230 1779 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 22:25:08.218322 1779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-0/wal
uuid: "75595c6cc3d54995ade109df7bbe8820"
format_stamp: "Formatted at 2026-08-08 22:24:47 on dist-test-slave-ncp9"
server_key: "fe802af7e6c84377e23aff40cce8d91c"
server_key_iv: "b670f24ecf1cfa8513b49cdb2376de4a"
server_key_version: "encryptionkey@0"
I20260808 22:25:08.218829 1779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:25:08.245008 1779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:25:08.248320 1779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:25:08.248507 1779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:25:08.249128 1779 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:25:08.250232 1837 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260808 22:25:08.254298 1779 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260808 22:25:08.254369 1779 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260808 22:25:08.254422 1779 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260808 22:25:08.256084 1779 ts_tablet_manager.cc:610] Registered 1 tablets
I20260808 22:25:08.256134 1779 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260808 22:25:08.256385 1837 tablet_bootstrap.cc:492] T e8c85cd9c93247edb3b4a8350db7a5ec P 75595c6cc3d54995ade109df7bbe8820: Bootstrap starting.
I20260808 22:25:08.266983 1779 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.193:34743
I20260808 22:25:08.267149 1985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.193:34743 every 8 connection(s)
I20260808 22:25:08.267975 1779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
I20260808 22:25:08.271996 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 1779
I20260808 22:25:08.272502 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.194:35133
--local_ip_for_outbound_sockets=127.9.176.194
--tserver_master_addrs=127.9.176.254:39769
--webserver_port=44931
--webserver_interface=127.9.176.194
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.194.keytab
--principal=kudu/127.9.176.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:33147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 22:25:08.276167 1837 log.cc:824] T e8c85cd9c93247edb3b4a8350db7a5ec P 75595c6cc3d54995ade109df7bbe8820: Log is configured to *not* fsync() on all Append() calls
Aug 08 22:25:08 dist-test-slave-ncp9 krb5kdc[860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227908, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.193@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
I20260808 22:25:08.285207 1837 tablet_bootstrap.cc:492] T e8c85cd9c93247edb3b4a8350db7a5ec P 75595c6cc3d54995ade109df7bbe8820: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 22:25:08.285440 1991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:08.270120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:39769 (local address 127.9.176.193:50001)
0808 22:25:08.271028 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:08.271119 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0808 22:25:08.272116 (+ 997us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:25:08.273605 (+ 1489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:25:08.273615 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:25:08.274100 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:25:08.274658 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:08.274673 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:08.276013 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:08.276016 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:08.277211 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:08.277222 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:08.277957 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:08.277962 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:08.278157 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:08.278739 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:25:08.278756 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:25:08.280232 (+ 1476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:25:08.282324 (+ 2092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:25:08.282330 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:25:08.282342 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:25:08.282615 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:25:08.282972 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:25:08.282976 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:25:08.282978 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:25:08.283126 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:25:08.283701 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:25:08.283708 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:25:08.283953 (+ 245us) client_negotiation.cc:769] Sending connection context
0808 22:25:08.284127 (+ 174us) client_negotiation.cc:240] Negotiation successful
0808 22:25:08.284460 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":152,"threads_started":1}
I20260808 22:25:08.285646 1837 tablet_bootstrap.cc:492] T e8c85cd9c93247edb3b4a8350db7a5ec P 75595c6cc3d54995ade109df7bbe8820: Bootstrap complete.
I20260808 22:25:08.285470 1992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:08.270242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.193:50001 (local address 127.9.176.254:39769)
0808 22:25:08.271265 (+ 1023us) server_negotiation.cc:206] Beginning negotiation
0808 22:25:08.271290 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0808 22:25:08.271337 (+ 47us) server_negotiation.cc:407] Connection header received
0808 22:25:08.272632 (+ 1295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:25:08.272649 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:25:08.273096 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:25:08.273510 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:25:08.274840 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:08.275833 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:08.277335 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:08.277809 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:08.277985 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:08.280427 (+ 2442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:25:08.280440 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:25:08.280452 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:25:08.280482 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:25:08.282198 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:25:08.282734 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:25:08.282740 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:25:08.282744 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:25:08.282868 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:25:08.283222 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:25:08.283226 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:25:08.283228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:25:08.283582 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:25:08.283715 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0808 22:25:08.284260 (+ 545us) server_negotiation.cc:299] Negotiation successful
0808 22:25:08.284535 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":879,"thread_start_us":133,"threads_started":1}
I20260808 22:25:08.287178 1837 ts_tablet_manager.cc:1397] T e8c85cd9c93247edb3b4a8350db7a5ec P 75595c6cc3d54995ade109df7bbe8820: Time spent bootstrapping tablet: real 0.031s user 0.020s sys 0.008s
I20260808 22:25:08.288003 1987 heartbeater.cc:347] Connected to a master server at 127.9.176.254:39769
I20260808 22:25:08.288237 1987 heartbeater.cc:464] Registering TS with master...
I20260808 22:25:08.288724 1987 heartbeater.cc:511] Master 127.9.176.254:39769 requested a full tablet report, sending...
I20260808 22:25:08.290107 1837 raft_consensus.cc:348] T e8c85cd9c93247edb3b4a8350db7a5ec P 75595c6cc3d54995ade109df7bbe8820 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75595c6cc3d54995ade109df7bbe8820" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34743 } } peers { permanent_uuid: "87604ec3fe8c43b688447d96bd4bfd3e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 35133 } } peers { permanent_uuid: "a606b1ce23654e70be927e2072afa5c6" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 35701 } }
I20260808 22:25:08.290623 1837 raft_consensus.cc:729] T e8c85cd9c93247edb3b4a8350db7a5ec P 75595c6cc3d54995ade109df7bbe8820 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75595c6cc3d54995ade109df7bbe8820, State: Initialized, Role: FOLLOWER
I20260808 22:25:08.291164 1727 ts_manager.cc:194] Registered new tserver with Master: 75595c6cc3d54995ade109df7bbe8820 (127.9.176.193:34743)
I20260808 22:25:08.291164 1837 consensus_queue.cc:260] T e8c85cd9c93247edb3b4a8350db7a5ec P 75595c6cc3d54995ade109df7bbe8820 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75595c6cc3d54995ade109df7bbe8820" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34743 } } peers { permanent_uuid: "87604ec3fe8c43b688447d96bd4bfd3e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 35133 } } peers { permanent_uuid: "a606b1ce23654e70be927e2072afa5c6" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 35701 } }
I20260808 22:25:08.292140 1837 ts_tablet_manager.cc:1428] T e8c85cd9c93247edb3b4a8350db7a5ec P 75595c6cc3d54995ade109df7bbe8820: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260808 22:25:08.294783 1727 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.176.193@KRBTEST.COM'} at 127.9.176.193:50001
I20260808 22:25:08.297348 1987 heartbeater.cc:503] Master 127.9.176.254:39769 was elected leader, sending a full tablet report...
Aug 08 22:25:08 dist-test-slave-ncp9 krb5kdc[860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227908, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.254@KRBTEST.COM for kudu/127.9.176.193@KRBTEST.COM
I20260808 22:25:08.305440 1997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:08.294151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.193:34743 (local address 127.0.0.1:59328)
0808 22:25:08.294587 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:08.294635 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0808 22:25:08.294760 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:25:08.295741 (+ 981us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:25:08.295748 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:25:08.296221 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:25:08.296359 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:08.296369 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:08.297806 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:08.297810 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:08.298718 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:08.298730 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:08.299476 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:08.299480 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:08.299565 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:08.300117 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:25:08.300131 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:25:08.301614 (+ 1483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:25:08.303751 (+ 2137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:25:08.303759 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:25:08.303764 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:25:08.303984 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:25:08.304283 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:25:08.304285 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:25:08.304286 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:25:08.304359 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:25:08.304860 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:25:08.304865 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:25:08.305032 (+ 167us) client_negotiation.cc:769] Sending connection context
0808 22:25:08.305160 (+ 128us) client_negotiation.cc:240] Negotiation successful
0808 22:25:08.305311 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":121,"threads_started":1}
I20260808 22:25:08.305610 1998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:08.294991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59328 (local address 127.9.176.193:34743)
0808 22:25:08.295288 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0808 22:25:08.295299 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 22:25:08.295320 (+ 21us) server_negotiation.cc:407] Connection header received
0808 22:25:08.295386 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:25:08.295391 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:25:08.295589 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:25:08.295718 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:25:08.296505 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:08.297681 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:08.298832 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:08.299317 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:08.299425 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:08.301766 (+ 2341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:25:08.301776 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:25:08.301788 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:25:08.301814 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:25:08.303638 (+ 1824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:25:08.304094 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:25:08.304100 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:25:08.304102 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:25:08.304199 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:25:08.304428 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:25:08.304431 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:25:08.304433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:25:08.304722 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:25:08.304855 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0808 22:25:08.305265 (+ 410us) server_negotiation.cc:299] Negotiation successful
0808 22:25:08.305437 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":7,"server-negotiator.queue_time_us":201,"thread_start_us":105,"threads_started":1}
I20260808 22:25:08.306552 1894 tablet_service.cc:1514] Processing DeleteTablet for tablet e8c85cd9c93247edb3b4a8350db7a5ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:24:47 UTC) from {username='kudu', principal='kudu/127.9.176.254@KRBTEST.COM'} at 127.0.0.1:59328
I20260808 22:25:08.307361 2000 tablet_replica.cc:330] stopping tablet replica
I20260808 22:25:08.307646 2000 raft_consensus.cc:2229] T e8c85cd9c93247edb3b4a8350db7a5ec P 75595c6cc3d54995ade109df7bbe8820 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:25:08.307864 2000 raft_consensus.cc:2258] T e8c85cd9c93247edb3b4a8350db7a5ec P 75595c6cc3d54995ade109df7bbe8820 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:25:08.308782 2000 ts_tablet_manager.cc:1905] T e8c85cd9c93247edb3b4a8350db7a5ec P 75595c6cc3d54995ade109df7bbe8820: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:25:08.312294 2000 ts_tablet_manager.cc:1918] T e8c85cd9c93247edb3b4a8350db7a5ec P 75595c6cc3d54995ade109df7bbe8820: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 22:25:08.312378 2000 log.cc:1196] T e8c85cd9c93247edb3b4a8350db7a5ec P 75595c6cc3d54995ade109df7bbe8820: Deleting WAL directory at /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-0/wal/wals/e8c85cd9c93247edb3b4a8350db7a5ec
I20260808 22:25:08.312641 2000 ts_tablet_manager.cc:1939] T e8c85cd9c93247edb3b4a8350db7a5ec P 75595c6cc3d54995ade109df7bbe8820: Deleting consensus metadata
I20260808 22:25:08.314307 1706 catalog_manager.cc:4841] TS 75595c6cc3d54995ade109df7bbe8820 (127.9.176.193:34743): tablet e8c85cd9c93247edb3b4a8350db7a5ec (table test-table [id=abfd36887513494aa55376a91bfc02ef]) successfully deleted
W20260808 22:25:08.392222 1993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:25:08.392477 1993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:25:08.392534 1993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:25:08.396648 1993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:25:08.396718 1993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:25:08.396798 1993 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.194
I20260808 22:25:08.402132 1993 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:33147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.194.keytab
--principal=kudu/127.9.176.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.194:35133
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.176.194
--webserver_port=44931
--tserver_master_addrs=127.9.176.254:39769
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1993
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.194
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:25:08.403378 1993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:25:08.404413 1993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:25:08.414036 2013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:25:08.414041 2006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:25:08.414251 1993 server_base.cc:1034] running on GCE node
W20260808 22:25:08.414386 2007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:25:08.414788 1993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:25:08.415452 1993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:25:08.416631 1993 hybrid_clock.cc:648] HybridClock initialized: now 1786227908416601 us; error 41 us; skew 500 ppm
Aug 08 22:25:08 dist-test-slave-ncp9 krb5kdc[860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227908, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:25:08.419345 1993 init.cc:377] Logged in from keytab as kudu/127.9.176.194@KRBTEST.COM (short username kudu)
I20260808 22:25:08.420256 1993 webserver.cc:460] Webserver started at http://127.9.176.194:44931/ using document root <none> and password file <none>
I20260808 22:25:08.420790 1993 fs_manager.cc:359] Metadata directory not provided
I20260808 22:25:08.420866 1993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:25:08.424824 1993 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260808 22:25:08.427004 2025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:25:08.428165 1993 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 22:25:08.428288 1993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-1/wal
uuid: "87604ec3fe8c43b688447d96bd4bfd3e"
format_stamp: "Formatted at 2026-08-08 22:24:47 on dist-test-slave-ncp9"
server_key: "4a57b74ff8dccf91150efac3fb9918d9"
server_key_iv: "2eb1172c80a8d90234e8baa2c54d6673"
server_key_version: "encryptionkey@0"
I20260808 22:25:08.428838 1993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:25:08.450428 1993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:25:08.453246 1993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:25:08.453423 1993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:25:08.454015 1993 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:25:08.454836 1993 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:25:08.454885 1993 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:25:08.454938 1993 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:25:08.454967 1993 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:25:08.463986 1993 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.194:35133
I20260808 22:25:08.464030 2202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.194:35133 every 8 connection(s)
I20260808 22:25:08.465008 1993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
I20260808 22:25:08.470118 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 1993
I20260808 22:25:08.470577 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.195:35701
--local_ip_for_outbound_sockets=127.9.176.195
--tserver_master_addrs=127.9.176.254:39769
--webserver_port=36629
--webserver_interface=127.9.176.195
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.195.keytab
--principal=kudu/127.9.176.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:33147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 22:25:08 dist-test-slave-ncp9 krb5kdc[860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227908, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.194@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
I20260808 22:25:08.478390 1992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:08.466814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.194:37729 (local address 127.9.176.254:39769)
0808 22:25:08.466945 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0808 22:25:08.466949 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:25:08.467791 (+ 842us) server_negotiation.cc:407] Connection header received
0808 22:25:08.469201 (+ 1410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:25:08.469204 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:25:08.469267 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:25:08.469341 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:25:08.470574 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:08.471171 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:08.472317 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:08.472561 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:08.472651 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:08.475043 (+ 2392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:25:08.475046 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:25:08.475047 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:25:08.475074 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:25:08.476406 (+ 1332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:25:08.476882 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:25:08.476885 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:25:08.476886 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:25:08.476997 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:25:08.477319 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:25:08.477322 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:25:08.477325 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:25:08.477504 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:25:08.477595 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0808 22:25:08.478084 (+ 489us) server_negotiation.cc:299] Negotiation successful
0808 22:25:08.478224 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260808 22:25:08.479185 2208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:08.467197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:39769 (local address 127.9.176.194:37729)
0808 22:25:08.467651 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:08.467695 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 22:25:08.468985 (+ 1290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:25:08.469533 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:25:08.469540 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:25:08.469896 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:25:08.470424 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:08.470441 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:08.471297 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:08.471300 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:08.472198 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:08.472208 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:08.472703 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:08.472707 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:08.472866 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:08.473494 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:25:08.473510 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:25:08.474872 (+ 1362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:25:08.476514 (+ 1642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:25:08.476520 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:25:08.476534 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:25:08.476786 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:25:08.477088 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:25:08.477091 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:25:08.477094 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:25:08.477230 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:25:08.477617 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:25:08.477624 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:25:08.477857 (+ 233us) client_negotiation.cc:769] Sending connection context
0808 22:25:08.478022 (+ 165us) client_negotiation.cc:240] Negotiation successful
0808 22:25:08.478351 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":111,"threads_started":1}
I20260808 22:25:08.480367 2203 heartbeater.cc:347] Connected to a master server at 127.9.176.254:39769
I20260808 22:25:08.480612 2203 heartbeater.cc:464] Registering TS with master...
I20260808 22:25:08.481138 2203 heartbeater.cc:511] Master 127.9.176.254:39769 requested a full tablet report, sending...
I20260808 22:25:08.482244 1727 ts_manager.cc:194] Registered new tserver with Master: 87604ec3fe8c43b688447d96bd4bfd3e (127.9.176.194:35133)
I20260808 22:25:08.482793 1727 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.176.194@KRBTEST.COM'} at 127.9.176.194:37729
W20260808 22:25:08.584017 2210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:25:08.584257 2210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:25:08.584316 2210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:25:08.588318 2210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:25:08.588379 2210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:25:08.588464 2210 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.195
I20260808 22:25:08.593712 2210 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:33147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.195.keytab
--principal=kudu/127.9.176.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.195:35701
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.176.195
--webserver_port=36629
--tserver_master_addrs=127.9.176.254:39769
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2210
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.195
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:25:08.594751 2210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:25:08.595705 2210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:25:08.606540 2220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:25:08.606581 2219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:25:08.606626 2223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:25:08.606886 2210 server_base.cc:1034] running on GCE node
I20260808 22:25:08.607338 2210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:25:08.607970 2210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:25:08.609154 2210 hybrid_clock.cc:648] HybridClock initialized: now 1786227908609138 us; error 30 us; skew 500 ppm
Aug 08 22:25:08 dist-test-slave-ncp9 krb5kdc[860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227908, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:25:08.612416 2210 init.cc:377] Logged in from keytab as kudu/127.9.176.195@KRBTEST.COM (short username kudu)
I20260808 22:25:08.613348 2210 webserver.cc:460] Webserver started at http://127.9.176.195:36629/ using document root <none> and password file <none>
I20260808 22:25:08.613883 2210 fs_manager.cc:359] Metadata directory not provided
I20260808 22:25:08.613956 2210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:25:08.617964 2210 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260808 22:25:08.619917 2232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:25:08.621163 2210 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 22:25:08.621279 2210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-2/wal
uuid: "a606b1ce23654e70be927e2072afa5c6"
format_stamp: "Formatted at 2026-08-08 22:24:47 on dist-test-slave-ncp9"
server_key: "11ee048a4c2fcedbcb20a9c6ecb83794"
server_key_iv: "d01037f3741c926609f3267f18ab274a"
server_key_version: "encryptionkey@0"
I20260808 22:25:08.621811 2210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:25:08.640486 2210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:25:08.643266 2210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:25:08.643443 2210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:25:08.644037 2210 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:25:08.644836 2210 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:25:08.644883 2210 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:25:08.644973 2210 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:25:08.645001 2210 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:25:08.653563 2210 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.195:35701
I20260808 22:25:08.653589 2405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.195:35701 every 8 connection(s)
I20260808 22:25:08.654479 2210 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
I20260808 22:25:08.657670 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 2210
Aug 08 22:25:08 dist-test-slave-ncp9 krb5kdc[860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227908, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.195@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
I20260808 22:25:08.666342 2413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:08.658676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41396 (local address 127.9.176.254:39769)
0808 22:25:08.658978 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0808 22:25:08.658982 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:25:08.658993 (+ 11us) server_negotiation.cc:407] Connection header received
0808 22:25:08.659029 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:25:08.659032 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:25:08.659072 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:25:08.659159 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:25:08.659884 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:08.660537 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:08.661502 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:08.661845 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:08.661973 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:08.663228 (+ 1255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:25:08.663232 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:25:08.663235 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:25:08.663257 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:25:08.664690 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:25:08.665117 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:25:08.665120 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:25:08.665121 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:25:08.665211 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:25:08.665487 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:25:08.665490 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:25:08.665492 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:25:08.665688 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:25:08.665881 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0808 22:25:08.666039 (+ 158us) server_negotiation.cc:299] Negotiation successful
0808 22:25:08.666172 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"spinlock_wait_cycles":6016,"thread_start_us":108,"threads_started":1}
I20260808 22:25:08.668577 1992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:08.656155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.195:45647 (local address 127.9.176.254:39769)
0808 22:25:08.656278 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0808 22:25:08.656282 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:25:08.657094 (+ 812us) server_negotiation.cc:407] Connection header received
0808 22:25:08.658575 (+ 1481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:25:08.658579 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:25:08.658635 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:25:08.658689 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:25:08.660189 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:08.660715 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:08.662137 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:08.662418 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:08.662521 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:08.665117 (+ 2596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:25:08.665120 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:25:08.665122 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:25:08.665166 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:25:08.666559 (+ 1393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:25:08.667121 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:25:08.667124 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:25:08.667125 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:25:08.667204 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:25:08.667543 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:25:08.667546 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:25:08.667548 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:25:08.667705 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:25:08.667786 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0808 22:25:08.668317 (+ 531us) server_negotiation.cc:299] Negotiation successful
0808 22:25:08.668424 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45,"spinlock_wait_cycles":768}
I20260808 22:25:08.669358 2409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:08.656491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:39769 (local address 127.9.176.195:45647)
0808 22:25:08.656953 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:08.657003 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0808 22:25:08.658325 (+ 1322us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:25:08.659116 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:25:08.659125 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:25:08.659504 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:25:08.660042 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:08.660061 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:08.660861 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:08.660864 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:08.662020 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:08.662030 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:08.662564 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:08.662567 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:08.662707 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:08.663217 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:25:08.663236 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:25:08.664969 (+ 1733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:25:08.666678 (+ 1709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:25:08.666682 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:25:08.666693 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:25:08.667007 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:25:08.667285 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:25:08.667287 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:25:08.667289 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:25:08.667418 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:25:08.667804 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:25:08.667811 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:25:08.668090 (+ 279us) client_negotiation.cc:769] Sending connection context
0808 22:25:08.668243 (+ 153us) client_negotiation.cc:240] Negotiation successful
0808 22:25:08.668545 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":114,"threads_started":1}
I20260808 22:25:08.670338 2406 heartbeater.cc:347] Connected to a master server at 127.9.176.254:39769
I20260808 22:25:08.670585 2406 heartbeater.cc:464] Registering TS with master...
I20260808 22:25:08.671036 2406 heartbeater.cc:511] Master 127.9.176.254:39769 requested a full tablet report, sending...
I20260808 22:25:08.671962 1727 ts_manager.cc:194] Registered new tserver with Master: a606b1ce23654e70be927e2072afa5c6 (127.9.176.195:35701)
I20260808 22:25:08.672474 1727 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.176.195@KRBTEST.COM'} at 127.9.176.195:45647
I20260808 22:25:08.679548 9923 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 22:25:08.683079 1992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:08.680113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41408 (local address 127.9.176.254:39769)
0808 22:25:08.680208 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0808 22:25:08.680211 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:25:08.680371 (+ 160us) server_negotiation.cc:407] Connection header received
0808 22:25:08.680523 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:25:08.680525 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:25:08.680612 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:25:08.680687 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:25:08.680829 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:08.681330 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:08.682162 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:08.682382 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:08.682456 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:08.682694 (+ 238us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:25:08.682809 (+ 115us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0808 22:25:08.682985 (+ 176us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41408: token expired
Metrics: {"server-negotiator.queue_time_us":36}
W20260808 22:25:08.683130 1992 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41408: token expired
W20260808 22:25:08.683450 1625 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.9.176.254:39769)
I20260808 22:25:08.689894 1992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:08.683768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41420 (local address 127.9.176.254:39769)
0808 22:25:08.683870 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0808 22:25:08.683873 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:25:08.683886 (+ 13us) server_negotiation.cc:407] Connection header received
0808 22:25:08.683985 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:25:08.683988 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:25:08.684036 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:25:08.684096 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:25:08.684645 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:08.685185 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:08.685928 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:08.686142 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:08.686221 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:08.687358 (+ 1137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:25:08.687362 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:25:08.687364 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:25:08.687387 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:25:08.688435 (+ 1048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:25:08.688823 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:25:08.688826 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:25:08.688828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:25:08.688932 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:25:08.689170 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:25:08.689173 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:25:08.689175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:25:08.689336 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:25:08.689447 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0808 22:25:08.689662 (+ 215us) server_negotiation.cc:299] Negotiation successful
0808 22:25:08.689763 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260808 22:25:08.692147 1727 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41420:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 22:25:08.695631 1727 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 22:25:08.703631 1894 tablet_service.cc:1467] Processing CreateTablet for tablet 3c5ec23c9759493aa1c33d7f0ea7e904 (DEFAULT_TABLE table=test-table [id=d8a4575a6ef8435ea3f2029dcb6281e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:25:08.704229 1894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c5ec23c9759493aa1c33d7f0ea7e904. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:25:08.706921 1997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:08.702126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.194:35133 (local address 127.0.0.1:46134)
0808 22:25:08.702259 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:08.702276 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 22:25:08.702358 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:25:08.703331 (+ 973us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:25:08.703334 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:25:08.703353 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:25:08.703465 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:08.703472 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:08.704466 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:08.704470 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:08.705493 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:08.705503 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:08.706489 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:08.706493 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:08.706589 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:08.706632 (+ 43us) client_negotiation.cc:769] Sending connection context
0808 22:25:08.706702 (+ 70us) client_negotiation.cc:240] Negotiation successful
0808 22:25:08.706787 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260808 22:25:08.707295 1837 tablet_bootstrap.cc:492] T 3c5ec23c9759493aa1c33d7f0ea7e904 P 75595c6cc3d54995ade109df7bbe8820: Bootstrap starting.
I20260808 22:25:08.708189 1837 tablet_bootstrap.cc:654] T 3c5ec23c9759493aa1c33d7f0ea7e904 P 75595c6cc3d54995ade109df7bbe8820: Neither blocks nor log segments found. Creating new log.
I20260808 22:25:08.709179 2424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:08.702334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46134 (local address 127.9.176.194:35133)
0808 22:25:08.702792 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0808 22:25:08.702806 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 22:25:08.702832 (+ 26us) server_negotiation.cc:407] Connection header received
0808 22:25:08.702920 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:25:08.702926 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:25:08.703166 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:25:08.703345 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:25:08.703551 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:08.704350 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:08.705640 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:08.706357 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:08.706469 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:08.706621 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0808 22:25:08.706756 (+ 135us) server_negotiation.cc:299] Negotiation successful
0808 22:25:08.706859 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":170,"threads_started":1}
I20260808 22:25:08.709987 2425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:08.702616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.195:35701 (local address 127.0.0.1:59360)
0808 22:25:08.702865 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:08.702876 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 22:25:08.702948 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:25:08.705865 (+ 2917us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:25:08.705869 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:25:08.705888 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:25:08.705986 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:08.705992 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:08.707246 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:08.707250 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:08.708104 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:08.708113 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:08.709615 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:08.709619 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:08.709706 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:08.709729 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 22:25:08.709800 (+ 71us) client_negotiation.cc:240] Negotiation successful
0808 22:25:08.709866 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":121,"threads_started":1}
I20260808 22:25:08.710111 2426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:08.703083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59360 (local address 127.9.176.195:35701)
0808 22:25:08.705249 (+ 2166us) server_negotiation.cc:206] Beginning negotiation
0808 22:25:08.705260 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 22:25:08.705283 (+ 23us) server_negotiation.cc:407] Connection header received
0808 22:25:08.705360 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:25:08.705365 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:25:08.705706 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:25:08.705877 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:25:08.706068 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:08.707152 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:08.708437 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:08.709287 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:08.709395 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:08.709551 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0808 22:25:08.709873 (+ 322us) server_negotiation.cc:299] Negotiation successful
0808 22:25:08.709959 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2059,"thread_start_us":92,"threads_started":1}
I20260808 22:25:08.710413 1837 tablet_bootstrap.cc:492] T 3c5ec23c9759493aa1c33d7f0ea7e904 P 75595c6cc3d54995ade109df7bbe8820: No bootstrap required, opened a new log
I20260808 22:25:08.710486 1837 ts_tablet_manager.cc:1397] T 3c5ec23c9759493aa1c33d7f0ea7e904 P 75595c6cc3d54995ade109df7bbe8820: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 22:25:08.710851 1837 raft_consensus.cc:348] T 3c5ec23c9759493aa1c33d7f0ea7e904 P 75595c6cc3d54995ade109df7bbe8820 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87604ec3fe8c43b688447d96bd4bfd3e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 35133 } } peers { permanent_uuid: "a606b1ce23654e70be927e2072afa5c6" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 35701 } } peers { permanent_uuid: "75595c6cc3d54995ade109df7bbe8820" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34743 } }
I20260808 22:25:08.710937 1837 raft_consensus.cc:374] T 3c5ec23c9759493aa1c33d7f0ea7e904 P 75595c6cc3d54995ade109df7bbe8820 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:25:08.710971 1837 raft_consensus.cc:729] T 3c5ec23c9759493aa1c33d7f0ea7e904 P 75595c6cc3d54995ade109df7bbe8820 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75595c6cc3d54995ade109df7bbe8820, State: Initialized, Role: FOLLOWER
I20260808 22:25:08.711079 1837 consensus_queue.cc:260] T 3c5ec23c9759493aa1c33d7f0ea7e904 P 75595c6cc3d54995ade109df7bbe8820 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87604ec3fe8c43b688447d96bd4bfd3e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 35133 } } peers { permanent_uuid: "a606b1ce23654e70be927e2072afa5c6" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 35701 } } peers { permanent_uuid: "75595c6cc3d54995ade109df7bbe8820" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34743 } }
I20260808 22:25:08.711351 1837 ts_tablet_manager.cc:1428] T 3c5ec23c9759493aa1c33d7f0ea7e904 P 75595c6cc3d54995ade109df7bbe8820: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260808 22:25:08.712448 2102 tablet_service.cc:1467] Processing CreateTablet for tablet 3c5ec23c9759493aa1c33d7f0ea7e904 (DEFAULT_TABLE table=test-table [id=d8a4575a6ef8435ea3f2029dcb6281e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:25:08.713646 2102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c5ec23c9759493aa1c33d7f0ea7e904. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:25:08.714447 2296 tablet_service.cc:1467] Processing CreateTablet for tablet 3c5ec23c9759493aa1c33d7f0ea7e904 (DEFAULT_TABLE table=test-table [id=d8a4575a6ef8435ea3f2029dcb6281e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:25:08.715485 2296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c5ec23c9759493aa1c33d7f0ea7e904. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:25:08.721982 2428 tablet_bootstrap.cc:492] T 3c5ec23c9759493aa1c33d7f0ea7e904 P 87604ec3fe8c43b688447d96bd4bfd3e: Bootstrap starting.
I20260808 22:25:08.723169 2430 tablet_bootstrap.cc:492] T 3c5ec23c9759493aa1c33d7f0ea7e904 P a606b1ce23654e70be927e2072afa5c6: Bootstrap starting.
I20260808 22:25:08.724537 2428 tablet_bootstrap.cc:654] T 3c5ec23c9759493aa1c33d7f0ea7e904 P 87604ec3fe8c43b688447d96bd4bfd3e: Neither blocks nor log segments found. Creating new log.
I20260808 22:25:08.725379 2430 tablet_bootstrap.cc:654] T 3c5ec23c9759493aa1c33d7f0ea7e904 P a606b1ce23654e70be927e2072afa5c6: Neither blocks nor log segments found. Creating new log.
I20260808 22:25:08.725406 2428 log.cc:824] T 3c5ec23c9759493aa1c33d7f0ea7e904 P 87604ec3fe8c43b688447d96bd4bfd3e: Log is configured to *not* fsync() on all Append() calls
I20260808 22:25:08.726126 2430 log.cc:824] T 3c5ec23c9759493aa1c33d7f0ea7e904 P a606b1ce23654e70be927e2072afa5c6: Log is configured to *not* fsync() on all Append() calls
I20260808 22:25:08.727305 2428 tablet_bootstrap.cc:492] T 3c5ec23c9759493aa1c33d7f0ea7e904 P 87604ec3fe8c43b688447d96bd4bfd3e: No bootstrap required, opened a new log
I20260808 22:25:08.727495 2428 ts_tablet_manager.cc:1397] T 3c5ec23c9759493aa1c33d7f0ea7e904 P 87604ec3fe8c43b688447d96bd4bfd3e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 22:25:08.727747 2430 tablet_bootstrap.cc:492] T 3c5ec23c9759493aa1c33d7f0ea7e904 P a606b1ce23654e70be927e2072afa5c6: No bootstrap required, opened a new log
I20260808 22:25:08.727908 2430 ts_tablet_manager.cc:1397] T 3c5ec23c9759493aa1c33d7f0ea7e904 P a606b1ce23654e70be927e2072afa5c6: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260808 22:25:08.730505 2428 raft_consensus.cc:348] T 3c5ec23c9759493aa1c33d7f0ea7e904 P 87604ec3fe8c43b688447d96bd4bfd3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87604ec3fe8c43b688447d96bd4bfd3e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 35133 } } peers { permanent_uuid: "a606b1ce23654e70be927e2072afa5c6" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 35701 } } peers { permanent_uuid: "75595c6cc3d54995ade109df7bbe8820" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34743 } }
I20260808 22:25:08.730739 2428 raft_consensus.cc:374] T 3c5ec23c9759493aa1c33d7f0ea7e904 P 87604ec3fe8c43b688447d96bd4bfd3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:25:08.730774 2428 raft_consensus.cc:729] T 3c5ec23c9759493aa1c33d7f0ea7e904 P 87604ec3fe8c43b688447d96bd4bfd3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87604ec3fe8c43b688447d96bd4bfd3e, State: Initialized, Role: FOLLOWER
I20260808 22:25:08.730888 2430 raft_consensus.cc:348] T 3c5ec23c9759493aa1c33d7f0ea7e904 P a606b1ce23654e70be927e2072afa5c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87604ec3fe8c43b688447d96bd4bfd3e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 35133 } } peers { permanent_uuid: "a606b1ce23654e70be927e2072afa5c6" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 35701 } } peers { permanent_uuid: "75595c6cc3d54995ade109df7bbe8820" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34743 } }
I20260808 22:25:08.731097 2430 raft_consensus.cc:374] T 3c5ec23c9759493aa1c33d7f0ea7e904 P a606b1ce23654e70be927e2072afa5c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:25:08.731158 2430 raft_consensus.cc:729] T 3c5ec23c9759493aa1c33d7f0ea7e904 P a606b1ce23654e70be927e2072afa5c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a606b1ce23654e70be927e2072afa5c6, State: Initialized, Role: FOLLOWER
I20260808 22:25:08.731205 2428 consensus_queue.cc:260] T 3c5ec23c9759493aa1c33d7f0ea7e904 P 87604ec3fe8c43b688447d96bd4bfd3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87604ec3fe8c43b688447d96bd4bfd3e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 35133 } } peers { permanent_uuid: "a606b1ce23654e70be927e2072afa5c6" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 35701 } } peers { permanent_uuid: "75595c6cc3d54995ade109df7bbe8820" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34743 } }
I20260808 22:25:08.731523 2430 consensus_queue.cc:260] T 3c5ec23c9759493aa1c33d7f0ea7e904 P a606b1ce23654e70be927e2072afa5c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87604ec3fe8c43b688447d96bd4bfd3e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 35133 } } peers { permanent_uuid: "a606b1ce23654e70be927e2072afa5c6" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 35701 } } peers { permanent_uuid: "75595c6cc3d54995ade109df7bbe8820" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34743 } }
I20260808 22:25:08.731868 2203 heartbeater.cc:503] Master 127.9.176.254:39769 was elected leader, sending a full tablet report...
I20260808 22:25:08.732178 2406 heartbeater.cc:503] Master 127.9.176.254:39769 was elected leader, sending a full tablet report...
I20260808 22:25:08.732440 2430 ts_tablet_manager.cc:1428] T 3c5ec23c9759493aa1c33d7f0ea7e904 P a606b1ce23654e70be927e2072afa5c6: Time spent starting tablet: real 0.004s user 0.004s sys 0.001s
I20260808 22:25:08.733436 2428 ts_tablet_manager.cc:1428] T 3c5ec23c9759493aa1c33d7f0ea7e904 P 87604ec3fe8c43b688447d96bd4bfd3e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260808 22:25:08.769204 1988 tablet.cc:2369] T 3c5ec23c9759493aa1c33d7f0ea7e904 P 75595c6cc3d54995ade109df7bbe8820: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 22:25:08.873970 2433 raft_consensus.cc:482] T 3c5ec23c9759493aa1c33d7f0ea7e904 P 87604ec3fe8c43b688447d96bd4bfd3e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 22:25:08.874195 2433 raft_consensus.cc:504] T 3c5ec23c9759493aa1c33d7f0ea7e904 P 87604ec3fe8c43b688447d96bd4bfd3e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87604ec3fe8c43b688447d96bd4bfd3e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 35133 } } peers { permanent_uuid: "a606b1ce23654e70be927e2072afa5c6" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 35701 } } peers { permanent_uuid: "75595c6cc3d54995ade109df7bbe8820" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34743 } }
I20260808 22:25:08.875281 2433 leader_election.cc:288] T 3c5ec23c9759493aa1c33d7f0ea7e904 P 87604ec3fe8c43b688447d96bd4bfd3e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a606b1ce23654e70be927e2072afa5c6 (127.9.176.195:35701), 75595c6cc3d54995ade109df7bbe8820 (127.9.176.193:34743)
I20260808 22:25:08.879954 2438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:08.875721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.194:59077 (local address 127.9.176.193:34743)
0808 22:25:08.875973 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0808 22:25:08.875977 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:25:08.876098 (+ 121us) server_negotiation.cc:407] Connection header received
0808 22:25:08.876222 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:25:08.876224 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:25:08.876277 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:25:08.876381 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:25:08.876615 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:08.877344 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:08.878708 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:08.879354 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:08.879454 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:08.879610 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0808 22:25:08.879754 (+ 144us) server_negotiation.cc:299] Negotiation successful
0808 22:25:08.879824 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":84,"threads_started":1}
I20260808 22:25:08.879951 2426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:08.875660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.194:41147 (local address 127.9.176.195:35701)
0808 22:25:08.875780 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0808 22:25:08.875784 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:25:08.875799 (+ 15us) server_negotiation.cc:407] Connection header received
0808 22:25:08.875965 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:25:08.875968 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:25:08.876032 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:25:08.876097 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:25:08.876383 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:08.877178 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:08.878715 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:08.879323 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:08.879449 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:08.879502 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0808 22:25:08.879756 (+ 254us) server_negotiation.cc:299] Negotiation successful
0808 22:25:08.879824 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 22:25:08.879977 2208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:08.875528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.195:35701 (local address 127.9.176.194:41147)
0808 22:25:08.875723 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:08.875737 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 22:25:08.875830 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:25:08.876139 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:25:08.876142 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:25:08.876157 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:25:08.876273 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:08.876278 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:08.877279 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:08.877281 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:08.878582 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:08.878594 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:08.879471 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:08.879475 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:08.879599 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:08.879645 (+ 46us) client_negotiation.cc:769] Sending connection context
0808 22:25:08.879732 (+ 87us) client_negotiation.cc:240] Negotiation successful
0808 22:25:08.879815 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":64}
I20260808 22:25:08.879980 2437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:08.875600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.193:34743 (local address 127.9.176.194:59077)
0808 22:25:08.876010 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:08.876022 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 22:25:08.876123 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:25:08.876406 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:25:08.876408 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:25:08.876422 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:25:08.876517 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:08.876521 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:08.877431 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:08.877434 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:08.878599 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:08.878607 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:08.879513 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:08.879516 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:08.879615 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:08.879669 (+ 54us) client_negotiation.cc:769] Sending connection context
0808 22:25:08.879732 (+ 63us) client_negotiation.cc:240] Negotiation successful
0808 22:25:08.879815 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"mutex_wait_us":45,"thread_start_us":126,"threads_started":1}
I20260808 22:25:08.880679 2327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c5ec23c9759493aa1c33d7f0ea7e904" candidate_uuid: "87604ec3fe8c43b688447d96bd4bfd3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a606b1ce23654e70be927e2072afa5c6" is_pre_election: true
I20260808 22:25:08.880667 1924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c5ec23c9759493aa1c33d7f0ea7e904" candidate_uuid: "87604ec3fe8c43b688447d96bd4bfd3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75595c6cc3d54995ade109df7bbe8820" is_pre_election: true
I20260808 22:25:08.880961 1924 raft_consensus.cc:2454] T 3c5ec23c9759493aa1c33d7f0ea7e904 P 75595c6cc3d54995ade109df7bbe8820 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87604ec3fe8c43b688447d96bd4bfd3e in term 0.
I20260808 22:25:08.880973 2327 raft_consensus.cc:2454] T 3c5ec23c9759493aa1c33d7f0ea7e904 P a606b1ce23654e70be927e2072afa5c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87604ec3fe8c43b688447d96bd4bfd3e in term 0.
I20260808 22:25:08.881394 2031 leader_election.cc:302] T 3c5ec23c9759493aa1c33d7f0ea7e904 P 87604ec3fe8c43b688447d96bd4bfd3e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75595c6cc3d54995ade109df7bbe8820, 87604ec3fe8c43b688447d96bd4bfd3e; no voters:
I20260808 22:25:08.881655 2433 raft_consensus.cc:2781] T 3c5ec23c9759493aa1c33d7f0ea7e904 P 87604ec3fe8c43b688447d96bd4bfd3e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 22:25:08.881738 2433 raft_consensus.cc:482] T 3c5ec23c9759493aa1c33d7f0ea7e904 P 87604ec3fe8c43b688447d96bd4bfd3e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 22:25:08.881771 2433 raft_consensus.cc:3037] T 3c5ec23c9759493aa1c33d7f0ea7e904 P 87604ec3fe8c43b688447d96bd4bfd3e [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:25:08.882793 2433 raft_consensus.cc:504] T 3c5ec23c9759493aa1c33d7f0ea7e904 P 87604ec3fe8c43b688447d96bd4bfd3e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87604ec3fe8c43b688447d96bd4bfd3e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 35133 } } peers { permanent_uuid: "a606b1ce23654e70be927e2072afa5c6" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 35701 } } peers { permanent_uuid: "75595c6cc3d54995ade109df7bbe8820" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34743 } }
I20260808 22:25:08.883126 2433 leader_election.cc:288] T 3c5ec23c9759493aa1c33d7f0ea7e904 P 87604ec3fe8c43b688447d96bd4bfd3e [CANDIDATE]: Term 1 election: Requested vote from peers a606b1ce23654e70be927e2072afa5c6 (127.9.176.195:35701), 75595c6cc3d54995ade109df7bbe8820 (127.9.176.193:34743)
I20260808 22:25:08.883430 2327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c5ec23c9759493aa1c33d7f0ea7e904" candidate_uuid: "87604ec3fe8c43b688447d96bd4bfd3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a606b1ce23654e70be927e2072afa5c6"
I20260808 22:25:08.883534 1924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c5ec23c9759493aa1c33d7f0ea7e904" candidate_uuid: "87604ec3fe8c43b688447d96bd4bfd3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75595c6cc3d54995ade109df7bbe8820"
I20260808 22:25:08.883566 2327 raft_consensus.cc:3037] T 3c5ec23c9759493aa1c33d7f0ea7e904 P a606b1ce23654e70be927e2072afa5c6 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:25:08.883661 1924 raft_consensus.cc:3037] T 3c5ec23c9759493aa1c33d7f0ea7e904 P 75595c6cc3d54995ade109df7bbe8820 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:25:08.884837 2327 raft_consensus.cc:2454] T 3c5ec23c9759493aa1c33d7f0ea7e904 P a606b1ce23654e70be927e2072afa5c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87604ec3fe8c43b688447d96bd4bfd3e in term 1.
I20260808 22:25:08.884837 1924 raft_consensus.cc:2454] T 3c5ec23c9759493aa1c33d7f0ea7e904 P 75595c6cc3d54995ade109df7bbe8820 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87604ec3fe8c43b688447d96bd4bfd3e in term 1.
I20260808 22:25:08.885135 2031 leader_election.cc:302] T 3c5ec23c9759493aa1c33d7f0ea7e904 P 87604ec3fe8c43b688447d96bd4bfd3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75595c6cc3d54995ade109df7bbe8820, 87604ec3fe8c43b688447d96bd4bfd3e; no voters:
I20260808 22:25:08.885305 2433 raft_consensus.cc:2781] T 3c5ec23c9759493aa1c33d7f0ea7e904 P 87604ec3fe8c43b688447d96bd4bfd3e [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:25:08.885529 2433 raft_consensus.cc:686] T 3c5ec23c9759493aa1c33d7f0ea7e904 P 87604ec3fe8c43b688447d96bd4bfd3e [term 1 LEADER]: Becoming Leader. State: Replica: 87604ec3fe8c43b688447d96bd4bfd3e, State: Running, Role: LEADER
I20260808 22:25:08.885860 2433 consensus_queue.cc:237] T 3c5ec23c9759493aa1c33d7f0ea7e904 P 87604ec3fe8c43b688447d96bd4bfd3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87604ec3fe8c43b688447d96bd4bfd3e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 35133 } } peers { permanent_uuid: "a606b1ce23654e70be927e2072afa5c6" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 35701 } } peers { permanent_uuid: "75595c6cc3d54995ade109df7bbe8820" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34743 } }
I20260808 22:25:08.888777 1727 catalog_manager.cc:5495] T 3c5ec23c9759493aa1c33d7f0ea7e904 P 87604ec3fe8c43b688447d96bd4bfd3e reported cstate change: term changed from 0 to 1, leader changed from <none> to 87604ec3fe8c43b688447d96bd4bfd3e (127.9.176.194). New cstate: current_term: 1 leader_uuid: "87604ec3fe8c43b688447d96bd4bfd3e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87604ec3fe8c43b688447d96bd4bfd3e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 35133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a606b1ce23654e70be927e2072afa5c6" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 35701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75595c6cc3d54995ade109df7bbe8820" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34743 } health_report { overall_health: UNKNOWN } } }
W20260808 22:25:08.905371 2407 tablet.cc:2369] T 3c5ec23c9759493aa1c33d7f0ea7e904 P a606b1ce23654e70be927e2072afa5c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 22:25:08.942962 2424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:08.939723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46138 (local address 127.9.176.194:35133)
0808 22:25:08.939850 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0808 22:25:08.939854 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:25:08.939867 (+ 13us) server_negotiation.cc:407] Connection header received
0808 22:25:08.939952 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:25:08.939954 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:25:08.940017 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:25:08.940103 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:25:08.940290 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:08.940785 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:08.941631 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:08.941861 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:08.941956 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:08.942215 (+ 259us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:25:08.942622 (+ 407us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:25:08.942715 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0808 22:25:08.942743 (+ 28us) server_negotiation.cc:299] Negotiation successful
0808 22:25:08.942795 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260808 22:25:08.948858 2327 raft_consensus.cc:1261] T 3c5ec23c9759493aa1c33d7f0ea7e904 P a606b1ce23654e70be927e2072afa5c6 [term 1 FOLLOWER]: Refusing update from remote peer 87604ec3fe8c43b688447d96bd4bfd3e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:25:08.948854 1924 raft_consensus.cc:1261] T 3c5ec23c9759493aa1c33d7f0ea7e904 P 75595c6cc3d54995ade109df7bbe8820 [term 1 FOLLOWER]: Refusing update from remote peer 87604ec3fe8c43b688447d96bd4bfd3e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:25:08.949615 2440 consensus_queue.cc:1035] T 3c5ec23c9759493aa1c33d7f0ea7e904 P 87604ec3fe8c43b688447d96bd4bfd3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a606b1ce23654e70be927e2072afa5c6" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 35701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:25:08.949790 2433 consensus_queue.cc:1035] T 3c5ec23c9759493aa1c33d7f0ea7e904 P 87604ec3fe8c43b688447d96bd4bfd3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "75595c6cc3d54995ade109df7bbe8820" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:25:08.958457 2447 mvcc.cc:205] Tried to move back new op lower bound from 7316389515047223296 to 7316389514798837760. Current Snapshot: MvccSnapshot[applied={T|T < 7316389515047223296}]
I20260808 22:25:08.963326 1725 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41420:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:25:08.963495 1725 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41420:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:25:08.965754 1725 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d2f7a8bd3d904d54b493540ee9062e2b: Sending DeleteTablet for 3 replicas of tablet 3c5ec23c9759493aa1c33d7f0ea7e904
I20260808 22:25:08.966372 2296 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c5ec23c9759493aa1c33d7f0ea7e904 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:25:08 UTC) from {username='kudu', principal='kudu/127.9.176.254@KRBTEST.COM'} at 127.0.0.1:59360
I20260808 22:25:08.966373 2102 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c5ec23c9759493aa1c33d7f0ea7e904 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:25:08 UTC) from {username='kudu', principal='kudu/127.9.176.254@KRBTEST.COM'} at 127.0.0.1:46134
I20260808 22:25:08.966782 1894 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c5ec23c9759493aa1c33d7f0ea7e904 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:25:08 UTC) from {username='kudu', principal='kudu/127.9.176.254@KRBTEST.COM'} at 127.0.0.1:59328
I20260808 22:25:08.966841 2455 tablet_replica.cc:330] T 3c5ec23c9759493aa1c33d7f0ea7e904 P a606b1ce23654e70be927e2072afa5c6: stopping tablet replica
I20260808 22:25:08.967068 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 1779
I20260808 22:25:08.967161 2455 raft_consensus.cc:2229] T 3c5ec23c9759493aa1c33d7f0ea7e904 P a606b1ce23654e70be927e2072afa5c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:25:08.967428 2455 raft_consensus.cc:2258] T 3c5ec23c9759493aa1c33d7f0ea7e904 P a606b1ce23654e70be927e2072afa5c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:25:08.973727 2459 tablet_replica.cc:330] T 3c5ec23c9759493aa1c33d7f0ea7e904 P 87604ec3fe8c43b688447d96bd4bfd3e: stopping tablet replica
I20260808 22:25:08.974067 2459 raft_consensus.cc:2229] T 3c5ec23c9759493aa1c33d7f0ea7e904 P 87604ec3fe8c43b688447d96bd4bfd3e [term 1 LEADER]: Raft consensus shutting down.
I20260808 22:25:08.974457 2459 raft_consensus.cc:2258] T 3c5ec23c9759493aa1c33d7f0ea7e904 P 87604ec3fe8c43b688447d96bd4bfd3e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:25:08.976243 2459 ts_tablet_manager.cc:1905] T 3c5ec23c9759493aa1c33d7f0ea7e904 P 87604ec3fe8c43b688447d96bd4bfd3e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:25:08.976243 2455 ts_tablet_manager.cc:1905] T 3c5ec23c9759493aa1c33d7f0ea7e904 P a606b1ce23654e70be927e2072afa5c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:25:08.979036 2455 ts_tablet_manager.cc:1918] T 3c5ec23c9759493aa1c33d7f0ea7e904 P a606b1ce23654e70be927e2072afa5c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 22:25:08.979144 2455 log.cc:1196] T 3c5ec23c9759493aa1c33d7f0ea7e904 P a606b1ce23654e70be927e2072afa5c6: Deleting WAL directory at /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-2/wal/wals/3c5ec23c9759493aa1c33d7f0ea7e904
I20260808 22:25:08.979218 2459 ts_tablet_manager.cc:1918] T 3c5ec23c9759493aa1c33d7f0ea7e904 P 87604ec3fe8c43b688447d96bd4bfd3e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 22:25:08.979362 2459 log.cc:1196] T 3c5ec23c9759493aa1c33d7f0ea7e904 P 87604ec3fe8c43b688447d96bd4bfd3e: Deleting WAL directory at /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-1/wal/wals/3c5ec23c9759493aa1c33d7f0ea7e904
I20260808 22:25:08.979509 2455 ts_tablet_manager.cc:1939] T 3c5ec23c9759493aa1c33d7f0ea7e904 P a606b1ce23654e70be927e2072afa5c6: Deleting consensus metadata
I20260808 22:25:08.979807 2459 ts_tablet_manager.cc:1939] T 3c5ec23c9759493aa1c33d7f0ea7e904 P 87604ec3fe8c43b688447d96bd4bfd3e: Deleting consensus metadata
I20260808 22:25:08.980839 1704 catalog_manager.cc:4841] TS a606b1ce23654e70be927e2072afa5c6 (127.9.176.195:35701): tablet 3c5ec23c9759493aa1c33d7f0ea7e904 (table test-table [id=d8a4575a6ef8435ea3f2029dcb6281e3]) successfully deleted
I20260808 22:25:08.980856 1707 catalog_manager.cc:4841] TS 87604ec3fe8c43b688447d96bd4bfd3e (127.9.176.194:35133): tablet 3c5ec23c9759493aa1c33d7f0ea7e904 (table test-table [id=d8a4575a6ef8435ea3f2029dcb6281e3]) successfully deleted
W20260808 22:25:08.982254 1706 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.176.193:34743 (error 108)
I20260808 22:25:08.982813 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 1993
I20260808 22:25:08.983304 2425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:08.982998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.193:34743 (local address 127.0.0.1:47136)
0808 22:25:08.983164 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:08.983227 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.193:34743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
W20260808 22:25:08.984598 1706 catalog_manager.cc:4584] TS 75595c6cc3d54995ade109df7bbe8820 (127.9.176.193:34743): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3c5ec23c9759493aa1c33d7f0ea7e904: Network error: Client connection negotiation failed: client connection to 127.9.176.193:34743: connect: Connection refused (error 111)
I20260808 22:25:08.990734 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 2210
I20260808 22:25:09.012384 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 1679
I20260808 22:25:29.023325 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:39769
--webserver_interface=127.9.176.254
--webserver_port=39755
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab
--principal=kudu/127.9.176.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:33147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.176.254:39769
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:25:29.133037 2462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:25:29.133288 2462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:25:29.133344 2462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:25:29.137580 2462 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260808 22:25:29.137629 2462 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260808 22:25:29.137650 2462 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260808 22:25:29.137668 2462 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:25:29.137701 2462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:25:29.137735 2462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:25:29.137763 2462 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:25:29.137790 2462 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:25:29.142983 2462 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:33147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.9.176.254:39769
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab
--principal=kudu/127.9.176.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:39769
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.176.254
--webserver_port=39755
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2462
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:25:29.143993 2462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:25:29.145013 2462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:25:29.153105 2470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:25:29.153131 2475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:25:29.153141 2471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:25:29.153227 2462 server_base.cc:1034] running on GCE node
I20260808 22:25:29.153831 2462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:25:29.154872 2462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:25:29.156039 2462 hybrid_clock.cc:648] HybridClock initialized: now 1786227929156010 us; error 38 us; skew 500 ppm
Aug 08 22:25:29 dist-test-slave-ncp9 krb5kdc[860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227929, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:25:29.158654 2462 init.cc:377] Logged in from keytab as kudu/127.9.176.254@KRBTEST.COM (short username kudu)
I20260808 22:25:29.159519 2462 webserver.cc:460] Webserver started at http://127.9.176.254:39755/ using document root <none> and password file <none>
I20260808 22:25:29.160030 2462 fs_manager.cc:359] Metadata directory not provided
I20260808 22:25:29.160096 2462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:25:29.163786 2462 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260808 22:25:29.165768 2490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:25:29.166909 2462 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 22:25:29.167034 2462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/master-0/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/master-0/wal
uuid: "d2f7a8bd3d904d54b493540ee9062e2b"
format_stamp: "Formatted at 2026-08-08 22:24:46 on dist-test-slave-ncp9"
server_key: "bf076ba4a10ac6e06aecea6b2951bf93"
server_key_iv: "416756c26c90b7cc146d954b67ace62e"
server_key_version: "encryptionkey@0"
I20260808 22:25:29.167536 2462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:25:29.180755 2462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:25:29.183437 2462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:25:29.183617 2462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:25:29.190539 2462 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.254:39769
I20260808 22:25:29.190795 2575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.254:39769 every 8 connection(s)
I20260808 22:25:29.191574 2462 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
I20260808 22:25:29.195703 2577 sys_catalog.cc:263] Verifying existing consensus state
I20260808 22:25:29.197224 2577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2f7a8bd3d904d54b493540ee9062e2b: Bootstrap starting.
I20260808 22:25:29.200160 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 2462
I20260808 22:25:29.200603 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.193:34743
--local_ip_for_outbound_sockets=127.9.176.193
--tserver_master_addrs=127.9.176.254:39769
--webserver_port=43873
--webserver_interface=127.9.176.193
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.193.keytab
--principal=kudu/127.9.176.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:33147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 22:25:29.215756 2577 log.cc:824] T 00000000000000000000000000000000 P d2f7a8bd3d904d54b493540ee9062e2b: Log is configured to *not* fsync() on all Append() calls
I20260808 22:25:29.227167 2577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2f7a8bd3d904d54b493540ee9062e2b: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 22:25:29.227535 2577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2f7a8bd3d904d54b493540ee9062e2b: Bootstrap complete.
I20260808 22:25:29.231494 2577 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2f7a8bd3d904d54b493540ee9062e2b [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2f7a8bd3d904d54b493540ee9062e2b" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 39769 } }
I20260808 22:25:29.231895 2577 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2f7a8bd3d904d54b493540ee9062e2b [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2f7a8bd3d904d54b493540ee9062e2b, State: Initialized, Role: FOLLOWER
I20260808 22:25:29.232380 2577 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2f7a8bd3d904d54b493540ee9062e2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2f7a8bd3d904d54b493540ee9062e2b" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 39769 } }
I20260808 22:25:29.232510 2577 raft_consensus.cc:388] T 00000000000000000000000000000000 P d2f7a8bd3d904d54b493540ee9062e2b [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:25:29.232576 2577 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2f7a8bd3d904d54b493540ee9062e2b [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:25:29.232671 2577 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2f7a8bd3d904d54b493540ee9062e2b [term 2 FOLLOWER]: Advancing to term 3
I20260808 22:25:29.234531 2577 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2f7a8bd3d904d54b493540ee9062e2b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2f7a8bd3d904d54b493540ee9062e2b" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 39769 } }
I20260808 22:25:29.234846 2577 leader_election.cc:302] T 00000000000000000000000000000000 P d2f7a8bd3d904d54b493540ee9062e2b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2f7a8bd3d904d54b493540ee9062e2b; no voters:
I20260808 22:25:29.235132 2577 leader_election.cc:288] T 00000000000000000000000000000000 P d2f7a8bd3d904d54b493540ee9062e2b [CANDIDATE]: Term 3 election: Requested vote from peers
I20260808 22:25:29.235272 2583 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2f7a8bd3d904d54b493540ee9062e2b [term 3 FOLLOWER]: Leader election won for term 3
I20260808 22:25:29.235514 2583 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2f7a8bd3d904d54b493540ee9062e2b [term 3 LEADER]: Becoming Leader. State: Replica: d2f7a8bd3d904d54b493540ee9062e2b, State: Running, Role: LEADER
I20260808 22:25:29.235877 2583 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2f7a8bd3d904d54b493540ee9062e2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2f7a8bd3d904d54b493540ee9062e2b" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 39769 } }
I20260808 22:25:29.236320 2577 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2f7a8bd3d904d54b493540ee9062e2b [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:25:29.237259 2586 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2f7a8bd3d904d54b493540ee9062e2b [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2f7a8bd3d904d54b493540ee9062e2b. Latest consensus state: current_term: 3 leader_uuid: "d2f7a8bd3d904d54b493540ee9062e2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2f7a8bd3d904d54b493540ee9062e2b" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 39769 } } }
I20260808 22:25:29.237385 2586 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2f7a8bd3d904d54b493540ee9062e2b [sys.catalog]: This master's current role is: LEADER
I20260808 22:25:29.238044 2585 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2f7a8bd3d904d54b493540ee9062e2b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "d2f7a8bd3d904d54b493540ee9062e2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2f7a8bd3d904d54b493540ee9062e2b" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 39769 } } }
I20260808 22:25:29.238137 2585 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2f7a8bd3d904d54b493540ee9062e2b [sys.catalog]: This master's current role is: LEADER
I20260808 22:25:29.238456 2594 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:25:29.243742 2594 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:25:29.244369 2594 catalog_manager.cc:1259] Loaded cluster ID: df74ca17b0754fd1913c4823f612329f
I20260808 22:25:29.244431 2594 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:25:29.247900 2594 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:25:29.248651 2594 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d2f7a8bd3d904d54b493540ee9062e2b: Loaded TSK: 0
I20260808 22:25:29.248734 2594 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d2f7a8bd3d904d54b493540ee9062e2b: Loaded TSK: 1
F20260808 22:25:29.249449 2594 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f90525a69ad google::LogMessage::Fail() at ??:0
@ 0x7f90525a988a google::LogMessage::SendToLog() at ??:0
@ 0x7f90525a64de google::LogMessage::Flush() at ??:0
@ 0x7f90525a7a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f905526a18d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f9055262f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f905cf0b7bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f905cee599c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f905cee6178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f905cf27df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f905cf4682b std::function<>::operator()() at ??:0
@ 0x7f905cee5b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f905cee6f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f905cee3579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f905cf27640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f90547f83d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f90547f8c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f90547fa3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f90547e9512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f9055e2d6db start_thread at ??:0
@ 0x7f90510a461f clone at ??:0
W20260808 22:25:29.348451 2581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:25:29.348688 2581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:25:29.348731 2581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:25:29.352700 2581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:25:29.352756 2581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:25:29.352816 2581 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.193
I20260808 22:25:29.358006 2581 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:33147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.193.keytab
--principal=kudu/127.9.176.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.193:34743
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.176.193
--webserver_port=43873
--tserver_master_addrs=127.9.176.254:39769
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.193
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:25:29.359023 2581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:25:29.360015 2581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:25:29.370405 2648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:25:29.370438 2649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:25:29.370471 2651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:25:29.370611 2581 server_base.cc:1034] running on GCE node
I20260808 22:25:29.371219 2581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:25:29.371814 2581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:25:29.372957 2581 hybrid_clock.cc:648] HybridClock initialized: now 1786227929372925 us; error 47 us; skew 500 ppm
Aug 08 22:25:29 dist-test-slave-ncp9 krb5kdc[860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227929, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:25:29.375614 2581 init.cc:377] Logged in from keytab as kudu/127.9.176.193@KRBTEST.COM (short username kudu)
I20260808 22:25:29.376464 2581 webserver.cc:460] Webserver started at http://127.9.176.193:43873/ using document root <none> and password file <none>
I20260808 22:25:29.377022 2581 fs_manager.cc:359] Metadata directory not provided
I20260808 22:25:29.377087 2581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:25:29.380882 2581 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260808 22:25:29.383080 2661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:25:29.384313 2581 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260808 22:25:29.384438 2581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-0/wal
uuid: "75595c6cc3d54995ade109df7bbe8820"
format_stamp: "Formatted at 2026-08-08 22:24:47 on dist-test-slave-ncp9"
server_key: "fe802af7e6c84377e23aff40cce8d91c"
server_key_iv: "b670f24ecf1cfa8513b49cdb2376de4a"
server_key_version: "encryptionkey@0"
I20260808 22:25:29.384997 2581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:25:29.415630 2581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:25:29.418989 2581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:25:29.419184 2581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:25:29.419804 2581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:25:29.420991 2669 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260808 22:25:29.426465 2581 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260808 22:25:29.426523 2581 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260808 22:25:29.426586 2581 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260808 22:25:29.428223 2581 ts_tablet_manager.cc:610] Registered 1 tablets
I20260808 22:25:29.428273 2581 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260808 22:25:29.428620 2669 tablet_bootstrap.cc:492] T 3c5ec23c9759493aa1c33d7f0ea7e904 P 75595c6cc3d54995ade109df7bbe8820: Bootstrap starting.
I20260808 22:25:29.438242 2581 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.193:34743
I20260808 22:25:29.438560 2835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.193:34743 every 8 connection(s)
I20260808 22:25:29.439227 2581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
I20260808 22:25:29.448832 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 2581
I20260808 22:25:29.449270 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.194:35133
--local_ip_for_outbound_sockets=127.9.176.194
--tserver_master_addrs=127.9.176.254:39769
--webserver_port=44931
--webserver_interface=127.9.176.194
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.194.keytab
--principal=kudu/127.9.176.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:33147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 22:25:29.451474 2669 log.cc:824] T 3c5ec23c9759493aa1c33d7f0ea7e904 P 75595c6cc3d54995ade109df7bbe8820: Log is configured to *not* fsync() on all Append() calls
I20260808 22:25:29.458714 2669 tablet_bootstrap.cc:492] T 3c5ec23c9759493aa1c33d7f0ea7e904 P 75595c6cc3d54995ade109df7bbe8820: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 22:25:29.459035 2669 tablet_bootstrap.cc:492] T 3c5ec23c9759493aa1c33d7f0ea7e904 P 75595c6cc3d54995ade109df7bbe8820: Bootstrap complete.
I20260808 22:25:29.460069 2669 ts_tablet_manager.cc:1397] T 3c5ec23c9759493aa1c33d7f0ea7e904 P 75595c6cc3d54995ade109df7bbe8820: Time spent bootstrapping tablet: real 0.032s user 0.017s sys 0.011s
I20260808 22:25:29.462441 2669 raft_consensus.cc:348] T 3c5ec23c9759493aa1c33d7f0ea7e904 P 75595c6cc3d54995ade109df7bbe8820 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87604ec3fe8c43b688447d96bd4bfd3e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 35133 } } peers { permanent_uuid: "a606b1ce23654e70be927e2072afa5c6" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 35701 } } peers { permanent_uuid: "75595c6cc3d54995ade109df7bbe8820" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34743 } }
I20260808 22:25:29.462867 2669 raft_consensus.cc:729] T 3c5ec23c9759493aa1c33d7f0ea7e904 P 75595c6cc3d54995ade109df7bbe8820 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75595c6cc3d54995ade109df7bbe8820, State: Initialized, Role: FOLLOWER
I20260808 22:25:29.463315 2669 consensus_queue.cc:260] T 3c5ec23c9759493aa1c33d7f0ea7e904 P 75595c6cc3d54995ade109df7bbe8820 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87604ec3fe8c43b688447d96bd4bfd3e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 35133 } } peers { permanent_uuid: "a606b1ce23654e70be927e2072afa5c6" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 35701 } } peers { permanent_uuid: "75595c6cc3d54995ade109df7bbe8820" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34743 } }
I20260808 22:25:29.464218 2669 ts_tablet_manager.cc:1428] T 3c5ec23c9759493aa1c33d7f0ea7e904 P 75595c6cc3d54995ade109df7bbe8820: Time spent starting tablet: real 0.004s user 0.007s sys 0.000s
W20260808 22:25:29.480101 2838 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:25:29.441313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:39769 (local address 127.9.176.193:50413)
0808 22:25:29.441846 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:29.441890 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 22:25:29.442753 (+ 863us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:25:29.479348 (+ 36595us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:39769: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":161,"threads_started":1}
W20260808 22:25:29.480552 2836 heartbeater.cc:644] Failed to heartbeat to 127.9.176.254:39769 (0 consecutive failures): Network error: Failed to ping master at 127.9.176.254:39769: Client connection negotiation failed: client connection to 127.9.176.254:39769: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260808 22:25:29.481252 2838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:29.481024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:39769 (local address 127.9.176.193:53623)
0808 22:25:29.481135 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:29.481201 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:39769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260808 22:25:29.481812 2838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:29.481627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:39769 (local address 127.9.176.193:51267)
0808 22:25:29.481721 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:29.481765 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:39769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260808 22:25:29.481953 2836 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 22:25:29.568876 2839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:25:29.569133 2839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:25:29.569191 2839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:25:29.573071 2839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:25:29.573119 2839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:25:29.573195 2839 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.194
I20260808 22:25:29.578890 2839 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:33147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.194.keytab
--principal=kudu/127.9.176.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.194:35133
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.176.194
--webserver_port=44931
--tserver_master_addrs=127.9.176.254:39769
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2839
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.194
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:25:29.579834 2839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:25:29.580798 2839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:25:29.590090 2853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:25:29.590142 2859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:25:29.590149 2855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:25:29.590231 2839 server_base.cc:1034] running on GCE node
I20260808 22:25:29.590699 2839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:25:29.591312 2839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:25:29.592464 2839 hybrid_clock.cc:648] HybridClock initialized: now 1786227929592440 us; error 41 us; skew 500 ppm
Aug 08 22:25:29 dist-test-slave-ncp9 krb5kdc[860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227929, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:25:29.598361 2839 init.cc:377] Logged in from keytab as kudu/127.9.176.194@KRBTEST.COM (short username kudu)
I20260808 22:25:29.599237 2839 webserver.cc:460] Webserver started at http://127.9.176.194:44931/ using document root <none> and password file <none>
I20260808 22:25:29.599757 2839 fs_manager.cc:359] Metadata directory not provided
I20260808 22:25:29.599825 2839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:25:29.603539 2839 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260808 22:25:29.605496 2867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:25:29.606642 2839 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 22:25:29.606760 2839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-1/wal
uuid: "87604ec3fe8c43b688447d96bd4bfd3e"
format_stamp: "Formatted at 2026-08-08 22:24:47 on dist-test-slave-ncp9"
server_key: "4a57b74ff8dccf91150efac3fb9918d9"
server_key_iv: "2eb1172c80a8d90234e8baa2c54d6673"
server_key_version: "encryptionkey@0"
I20260808 22:25:29.607276 2839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:25:29.628602 2839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:25:29.631289 2839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:25:29.631469 2839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:25:29.632081 2839 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:25:29.632925 2839 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:25:29.632978 2839 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:25:29.633033 2839 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:25:29.633059 2839 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:25:29.641791 2839 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.194:35133
I20260808 22:25:29.641825 3039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.194:35133 every 8 connection(s)
I20260808 22:25:29.642776 2839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
I20260808 22:25:29.646487 3042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:29.644756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:39769 (local address 127.9.176.194:58783)
0808 22:25:29.645224 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:29.645327 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:39769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":145,"threads_started":1}
I20260808 22:25:29.646872 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 2839
W20260808 22:25:29.646915 3040 heartbeater.cc:644] Failed to heartbeat to 127.9.176.254:39769 (0 consecutive failures): Network error: Failed to ping master at 127.9.176.254:39769: Client connection negotiation failed: client connection to 127.9.176.254:39769: connect: Connection refused (error 111)
I20260808 22:25:29.647336 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.195:35701
--local_ip_for_outbound_sockets=127.9.176.195
--tserver_master_addrs=127.9.176.254:39769
--webserver_port=36629
--webserver_interface=127.9.176.195
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.195.keytab
--principal=kudu/127.9.176.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:33147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 22:25:29.647491 3042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:29.647275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:39769 (local address 127.9.176.194:55593)
0808 22:25:29.647380 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:29.647428 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:39769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260808 22:25:29.648103 3042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:29.647877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:39769 (local address 127.9.176.194:38455)
0808 22:25:29.647976 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:29.648030 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:39769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260808 22:25:29.648293 3040 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 22:25:29.758224 3043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:25:29.758479 3043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:25:29.758519 3043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:25:29.762445 3043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:25:29.762502 3043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:25:29.762562 3043 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.195
I20260808 22:25:29.767808 3043 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:33147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.195.keytab
--principal=kudu/127.9.176.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.195:35701
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.176.195
--webserver_port=36629
--tserver_master_addrs=127.9.176.254:39769
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3043
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.195
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:25:29.768795 3043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:25:29.769800 3043 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:25:29.778935 3051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:25:29.778995 3055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:25:29.778959 3050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:25:29.779177 3043 server_base.cc:1034] running on GCE node
I20260808 22:25:29.779560 3043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:25:29.780180 3043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:25:29.781327 3043 hybrid_clock.cc:648] HybridClock initialized: now 1786227929781299 us; error 41 us; skew 500 ppm
Aug 08 22:25:29 dist-test-slave-ncp9 krb5kdc[860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227929, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:25:29.783882 3043 init.cc:377] Logged in from keytab as kudu/127.9.176.195@KRBTEST.COM (short username kudu)
I20260808 22:25:29.784677 3043 webserver.cc:460] Webserver started at http://127.9.176.195:36629/ using document root <none> and password file <none>
I20260808 22:25:29.785224 3043 fs_manager.cc:359] Metadata directory not provided
I20260808 22:25:29.785290 3043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:25:29.788883 3043 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260808 22:25:29.790853 3063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:25:29.791983 3043 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 22:25:29.792086 3043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-2/wal
uuid: "a606b1ce23654e70be927e2072afa5c6"
format_stamp: "Formatted at 2026-08-08 22:24:47 on dist-test-slave-ncp9"
server_key: "11ee048a4c2fcedbcb20a9c6ecb83794"
server_key_iv: "d01037f3741c926609f3267f18ab274a"
server_key_version: "encryptionkey@0"
I20260808 22:25:29.792600 3043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:25:29.805945 3043 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:25:29.808487 3043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:25:29.808670 3043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:25:29.809288 3043 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:25:29.810120 3043 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:25:29.810168 3043 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:25:29.810220 3043 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:25:29.810245 3043 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:25:29.818727 3043 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.195:35701
I20260808 22:25:29.818785 3214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.195:35701 every 8 connection(s)
I20260808 22:25:29.819689 3043 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
I20260808 22:25:29.823374 3221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:29.821694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:39769 (local address 127.9.176.195:52991)
0808 22:25:29.822149 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:29.822265 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:39769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":117,"threads_started":1}
W20260808 22:25:29.823887 3218 heartbeater.cc:644] Failed to heartbeat to 127.9.176.254:39769 (0 consecutive failures): Network error: Failed to ping master at 127.9.176.254:39769: Client connection negotiation failed: client connection to 127.9.176.254:39769: connect: Connection refused (error 111)
I20260808 22:25:29.824108 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 3043
I20260808 22:25:29.824473 3221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:29.824265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:39769 (local address 127.9.176.195:40763)
0808 22:25:29.824364 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:29.824413 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:39769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260808 22:25:29.825079 3221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:29.824867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:39769 (local address 127.9.176.195:52195)
0808 22:25:29.824989 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:29.825032 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:39769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260808 22:25:29.825239 3218 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260808 22:25:30.482995 3223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:30.482542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:39769 (local address 127.9.176.193:53577)
0808 22:25:30.482833 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:30.482891 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:39769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":86,"threads_started":1}
I20260808 22:25:30.650153 3224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:30.648833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:39769 (local address 127.9.176.194:51477)
0808 22:25:30.650001 (+ 1168us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:30.650054 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:39769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1095,"thread_start_us":1018,"threads_started":1}
I20260808 22:25:30.678500 3225 raft_consensus.cc:482] T 3c5ec23c9759493aa1c33d7f0ea7e904 P 75595c6cc3d54995ade109df7bbe8820 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 22:25:30.678763 3225 raft_consensus.cc:504] T 3c5ec23c9759493aa1c33d7f0ea7e904 P 75595c6cc3d54995ade109df7bbe8820 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87604ec3fe8c43b688447d96bd4bfd3e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 35133 } } peers { permanent_uuid: "a606b1ce23654e70be927e2072afa5c6" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 35701 } } peers { permanent_uuid: "75595c6cc3d54995ade109df7bbe8820" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34743 } }
I20260808 22:25:30.680083 3225 leader_election.cc:288] T 3c5ec23c9759493aa1c33d7f0ea7e904 P 75595c6cc3d54995ade109df7bbe8820 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 87604ec3fe8c43b688447d96bd4bfd3e (127.9.176.194:35133), a606b1ce23654e70be927e2072afa5c6 (127.9.176.195:35701)
Aug 08 22:25:30 dist-test-slave-ncp9 krb5kdc[860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227929, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.193@KRBTEST.COM for kudu/127.9.176.195@KRBTEST.COM
Aug 08 22:25:30 dist-test-slave-ncp9 krb5kdc[860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227929, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.193@KRBTEST.COM for kudu/127.9.176.194@KRBTEST.COM
I20260808 22:25:30.694113 3226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:30.680348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.195:35701 (local address 127.9.176.193:54045)
0808 22:25:30.680876 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:30.680912 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0808 22:25:30.681017 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:25:30.682558 (+ 1541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:25:30.682561 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:25:30.683066 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:25:30.683554 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:30.683569 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:30.684842 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:30.684845 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:30.685866 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:30.685876 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:30.686546 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:30.686549 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:30.686787 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:30.687284 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:25:30.687300 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:25:30.689190 (+ 1890us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:25:30.691234 (+ 2044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:25:30.691240 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:25:30.691253 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:25:30.691522 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:25:30.692200 (+ 678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:25:30.692203 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:25:30.692205 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:25:30.692295 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:25:30.693010 (+ 715us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:25:30.693017 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:25:30.693361 (+ 344us) client_negotiation.cc:769] Sending connection context
0808 22:25:30.693578 (+ 217us) client_negotiation.cc:240] Negotiation successful
0808 22:25:30.693915 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"mutex_wait_us":144,"spinlock_wait_cycles":31360,"thread_start_us":73,"threads_started":1}
I20260808 22:25:30.694257 3227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:30.680447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.193:46305 (local address 127.9.176.194:35133)
0808 22:25:30.680781 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0808 22:25:30.680808 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0808 22:25:30.680846 (+ 38us) server_negotiation.cc:407] Connection header received
0808 22:25:30.681772 (+ 926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:25:30.681779 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:25:30.682125 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:25:30.682442 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:25:30.683724 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:30.684705 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:30.685991 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:30.686460 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:30.686616 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:30.689395 (+ 2779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:25:30.689399 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:25:30.689409 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:25:30.689447 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:25:30.691254 (+ 1807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:25:30.691709 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:25:30.691715 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:25:30.691719 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:25:30.691840 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:25:30.692288 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:25:30.692291 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:25:30.692293 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:25:30.692859 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:25:30.693036 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0808 22:25:30.693748 (+ 712us) server_negotiation.cc:299] Negotiation successful
0808 22:25:30.694050 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":116,"threads_started":1}
I20260808 22:25:30.694257 3230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:30.680796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.193:54045 (local address 127.9.176.195:35701)
0808 22:25:30.681257 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0808 22:25:30.681271 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 22:25:30.681313 (+ 42us) server_negotiation.cc:407] Connection header received
0808 22:25:30.682015 (+ 702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:25:30.682021 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:25:30.682346 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:25:30.682594 (+ 248us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:25:30.683721 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:30.684696 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:30.685978 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:30.686431 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:30.686583 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:30.689384 (+ 2801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:25:30.689389 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:25:30.689400 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:25:30.689427 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:25:30.691120 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:25:30.691645 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:25:30.691650 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:25:30.691653 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:25:30.691792 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:25:30.692414 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:25:30.692417 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:25:30.692419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:25:30.692859 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:25:30.693036 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0808 22:25:30.693748 (+ 712us) server_negotiation.cc:299] Negotiation successful
0808 22:25:30.694050 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":164,"threads_started":1}
I20260808 22:25:30.694119 3223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:30.680278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.194:35133 (local address 127.9.176.193:46305)
0808 22:25:30.680383 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:30.680399 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 22:25:30.680510 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:25:30.682510 (+ 2000us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:25:30.682516 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:25:30.683065 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:25:30.683555 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:30.683569 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:30.684839 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:30.684843 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:30.685869 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:30.685880 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:30.686582 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:30.686587 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:30.686792 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:30.687436 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:25:30.687451 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:25:30.689187 (+ 1736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:25:30.691369 (+ 2182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:25:30.691373 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:25:30.691375 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:25:30.691598 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:25:30.691967 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:25:30.691970 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:25:30.691972 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:25:30.692151 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:25:30.693010 (+ 859us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:25:30.693017 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:25:30.693361 (+ 344us) client_negotiation.cc:769] Sending connection context
0808 22:25:30.693578 (+ 217us) client_negotiation.cc:240] Negotiation successful
0808 22:25:30.693979 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":397,"spinlock_wait_cycles":39808}
W20260808 22:25:30.696961 2665 leader_election.cc:341] T 3c5ec23c9759493aa1c33d7f0ea7e904 P 75595c6cc3d54995ade109df7bbe8820 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 87604ec3fe8c43b688447d96bd4bfd3e (127.9.176.194:35133): Not found: Tablet not found: 3c5ec23c9759493aa1c33d7f0ea7e904
W20260808 22:25:30.697103 2662 leader_election.cc:341] T 3c5ec23c9759493aa1c33d7f0ea7e904 P 75595c6cc3d54995ade109df7bbe8820 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a606b1ce23654e70be927e2072afa5c6 (127.9.176.195:35701): Not found: Tablet not found: 3c5ec23c9759493aa1c33d7f0ea7e904
I20260808 22:25:30.697175 2662 leader_election.cc:302] T 3c5ec23c9759493aa1c33d7f0ea7e904 P 75595c6cc3d54995ade109df7bbe8820 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 75595c6cc3d54995ade109df7bbe8820; no voters: 87604ec3fe8c43b688447d96bd4bfd3e, a606b1ce23654e70be927e2072afa5c6
I20260808 22:25:30.697389 3225 raft_consensus.cc:2726] T 3c5ec23c9759493aa1c33d7f0ea7e904 P 75595c6cc3d54995ade109df7bbe8820 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 22:25:30.826104 3231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:30.825705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:39769 (local address 127.9.176.195:39739)
0808 22:25:30.825939 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:30.825992 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:39769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":85,"threads_started":1}
I20260808 22:25:31.484059 3233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:31.483682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:39769 (local address 127.9.176.193:48593)
0808 22:25:31.483915 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:31.483967 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:39769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":75,"threads_started":1}
I20260808 22:25:31.651127 3234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:31.650748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:39769 (local address 127.9.176.194:44793)
0808 22:25:31.650987 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:31.651039 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:39769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":70,"threads_started":1}
I20260808 22:25:31.827191 3235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:31.826766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:39769 (local address 127.9.176.195:46329)
0808 22:25:31.827036 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:31.827088 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:39769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":80,"threads_started":1}
I20260808 22:25:32.485220 3237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:32.484768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:39769 (local address 127.9.176.193:45237)
0808 22:25:32.485051 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:32.485122 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:39769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":115,"threads_started":1}
I20260808 22:25:32.490828 3238 raft_consensus.cc:482] T 3c5ec23c9759493aa1c33d7f0ea7e904 P 75595c6cc3d54995ade109df7bbe8820 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 22:25:32.490947 3238 raft_consensus.cc:504] T 3c5ec23c9759493aa1c33d7f0ea7e904 P 75595c6cc3d54995ade109df7bbe8820 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87604ec3fe8c43b688447d96bd4bfd3e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 35133 } } peers { permanent_uuid: "a606b1ce23654e70be927e2072afa5c6" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 35701 } } peers { permanent_uuid: "75595c6cc3d54995ade109df7bbe8820" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34743 } }
I20260808 22:25:32.491298 3238 leader_election.cc:288] T 3c5ec23c9759493aa1c33d7f0ea7e904 P 75595c6cc3d54995ade109df7bbe8820 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 87604ec3fe8c43b688447d96bd4bfd3e (127.9.176.194:35133), a606b1ce23654e70be927e2072afa5c6 (127.9.176.195:35701)
W20260808 22:25:32.492066 2665 leader_election.cc:341] T 3c5ec23c9759493aa1c33d7f0ea7e904 P 75595c6cc3d54995ade109df7bbe8820 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 87604ec3fe8c43b688447d96bd4bfd3e (127.9.176.194:35133): Not found: Tablet not found: 3c5ec23c9759493aa1c33d7f0ea7e904
W20260808 22:25:32.492187 2662 leader_election.cc:341] T 3c5ec23c9759493aa1c33d7f0ea7e904 P 75595c6cc3d54995ade109df7bbe8820 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a606b1ce23654e70be927e2072afa5c6 (127.9.176.195:35701): Not found: Tablet not found: 3c5ec23c9759493aa1c33d7f0ea7e904
I20260808 22:25:32.492276 2662 leader_election.cc:302] T 3c5ec23c9759493aa1c33d7f0ea7e904 P 75595c6cc3d54995ade109df7bbe8820 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 75595c6cc3d54995ade109df7bbe8820; no voters: 87604ec3fe8c43b688447d96bd4bfd3e, a606b1ce23654e70be927e2072afa5c6
I20260808 22:25:32.492412 3238 raft_consensus.cc:2726] T 3c5ec23c9759493aa1c33d7f0ea7e904 P 75595c6cc3d54995ade109df7bbe8820 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 22:25:32.652243 3240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:32.651822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:39769 (local address 127.9.176.194:59391)
0808 22:25:32.652089 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:32.652149 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:39769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":75,"threads_started":1}
I20260808 22:25:32.828246 3244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:32.827858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:39769 (local address 127.9.176.195:44847)
0808 22:25:32.828095 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:32.828155 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:39769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":81,"threads_started":1}
I20260808 22:25:33.486294 3245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:33.485896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:39769 (local address 127.9.176.193:34165)
0808 22:25:33.486142 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:33.486194 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:39769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":90,"threads_started":1}
I20260808 22:25:33.653460 3246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:33.653019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:39769 (local address 127.9.176.194:48597)
0808 22:25:33.653297 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:33.653359 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:39769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":89,"threads_started":1}
I20260808 22:25:33.829429 3248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:33.828980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:39769 (local address 127.9.176.195:40753)
0808 22:25:33.829258 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:33.829324 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:39769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":85,"threads_started":1}
I20260808 22:25:34.487546 3249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:34.487128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:39769 (local address 127.9.176.193:59781)
0808 22:25:34.487395 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:34.487448 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:39769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":74,"threads_started":1}
I20260808 22:25:34.654546 3250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:34.654167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:39769 (local address 127.9.176.194:45313)
0808 22:25:34.654408 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:34.654455 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:39769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":78,"threads_started":1}
I20260808 22:25:34.830580 3251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:34.830196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:39769 (local address 127.9.176.195:57231)
0808 22:25:34.830426 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:34.830483 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:39769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":73,"threads_started":1}
I20260808 22:25:35.424721 3252 raft_consensus.cc:482] T 3c5ec23c9759493aa1c33d7f0ea7e904 P 75595c6cc3d54995ade109df7bbe8820 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 22:25:35.424849 3252 raft_consensus.cc:504] T 3c5ec23c9759493aa1c33d7f0ea7e904 P 75595c6cc3d54995ade109df7bbe8820 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87604ec3fe8c43b688447d96bd4bfd3e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 35133 } } peers { permanent_uuid: "a606b1ce23654e70be927e2072afa5c6" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 35701 } } peers { permanent_uuid: "75595c6cc3d54995ade109df7bbe8820" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34743 } }
I20260808 22:25:35.425293 3252 leader_election.cc:288] T 3c5ec23c9759493aa1c33d7f0ea7e904 P 75595c6cc3d54995ade109df7bbe8820 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 87604ec3fe8c43b688447d96bd4bfd3e (127.9.176.194:35133), a606b1ce23654e70be927e2072afa5c6 (127.9.176.195:35701)
W20260808 22:25:35.426059 2665 leader_election.cc:341] T 3c5ec23c9759493aa1c33d7f0ea7e904 P 75595c6cc3d54995ade109df7bbe8820 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 87604ec3fe8c43b688447d96bd4bfd3e (127.9.176.194:35133): Not found: Tablet not found: 3c5ec23c9759493aa1c33d7f0ea7e904
W20260808 22:25:35.426184 2662 leader_election.cc:341] T 3c5ec23c9759493aa1c33d7f0ea7e904 P 75595c6cc3d54995ade109df7bbe8820 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a606b1ce23654e70be927e2072afa5c6 (127.9.176.195:35701): Not found: Tablet not found: 3c5ec23c9759493aa1c33d7f0ea7e904
I20260808 22:25:35.426275 2662 leader_election.cc:302] T 3c5ec23c9759493aa1c33d7f0ea7e904 P 75595c6cc3d54995ade109df7bbe8820 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 75595c6cc3d54995ade109df7bbe8820; no voters: 87604ec3fe8c43b688447d96bd4bfd3e, a606b1ce23654e70be927e2072afa5c6
I20260808 22:25:35.426417 3252 raft_consensus.cc:2726] T 3c5ec23c9759493aa1c33d7f0ea7e904 P 75595c6cc3d54995ade109df7bbe8820 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 22:25:35.488643 3253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:35.488230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:39769 (local address 127.9.176.193:52427)
0808 22:25:35.488473 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:35.488538 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:39769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":74,"threads_started":1}
I20260808 22:25:35.655797 3256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:35.655362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:39769 (local address 127.9.176.194:46205)
0808 22:25:35.655637 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:35.655699 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:39769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":92,"threads_started":1}
I20260808 22:25:35.831688 3257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:35.831269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:39769 (local address 127.9.176.195:43113)
0808 22:25:35.831539 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:35.831594 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:39769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":86,"threads_started":1}
I20260808 22:25:36.489828 3259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:36.489419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:39769 (local address 127.9.176.193:54959)
0808 22:25:36.489676 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:36.489731 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:39769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":91,"threads_started":1}
I20260808 22:25:36.657066 3260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:36.656570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:39769 (local address 127.9.176.194:33631)
0808 22:25:36.656887 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:36.656971 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:39769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":95,"threads_started":1}
I20260808 22:25:36.832801 3262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:36.832385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:39769 (local address 127.9.176.195:45987)
0808 22:25:36.832660 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:36.832712 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:39769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":74,"threads_started":1}
I20260808 22:25:37.490906 3263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:37.490498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:39769 (local address 127.9.176.193:48655)
0808 22:25:37.490752 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:37.490808 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:39769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":80,"threads_started":1}
I20260808 22:25:37.658277 3265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:37.657808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:39769 (local address 127.9.176.194:36901)
0808 22:25:37.658122 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:37.658180 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:39769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":88,"threads_started":1}
I20260808 22:25:37.833894 3266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:37.833493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:39769 (local address 127.9.176.195:37493)
0808 22:25:37.833746 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:37.833800 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:39769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":83,"threads_started":1}
I20260808 22:25:38.492110 3267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:38.491666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:39769 (local address 127.9.176.193:40485)
0808 22:25:38.491952 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:38.492011 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:39769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":100,"threads_started":1}
I20260808 22:25:38.659391 3268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:38.658991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:39769 (local address 127.9.176.194:38405)
0808 22:25:38.659241 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:38.659298 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:39769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":90,"threads_started":1}
I20260808 22:25:38.835000 3272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:38.834571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:39769 (local address 127.9.176.195:59133)
0808 22:25:38.834844 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:38.834903 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:39769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":77,"threads_started":1}
I20260808 22:25:38.903555 3273 raft_consensus.cc:482] T 3c5ec23c9759493aa1c33d7f0ea7e904 P 75595c6cc3d54995ade109df7bbe8820 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 22:25:38.903718 3273 raft_consensus.cc:504] T 3c5ec23c9759493aa1c33d7f0ea7e904 P 75595c6cc3d54995ade109df7bbe8820 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87604ec3fe8c43b688447d96bd4bfd3e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 35133 } } peers { permanent_uuid: "a606b1ce23654e70be927e2072afa5c6" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 35701 } } peers { permanent_uuid: "75595c6cc3d54995ade109df7bbe8820" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34743 } }
I20260808 22:25:38.904178 3273 leader_election.cc:288] T 3c5ec23c9759493aa1c33d7f0ea7e904 P 75595c6cc3d54995ade109df7bbe8820 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 87604ec3fe8c43b688447d96bd4bfd3e (127.9.176.194:35133), a606b1ce23654e70be927e2072afa5c6 (127.9.176.195:35701)
W20260808 22:25:38.905001 2662 leader_election.cc:341] T 3c5ec23c9759493aa1c33d7f0ea7e904 P 75595c6cc3d54995ade109df7bbe8820 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a606b1ce23654e70be927e2072afa5c6 (127.9.176.195:35701): Not found: Tablet not found: 3c5ec23c9759493aa1c33d7f0ea7e904
W20260808 22:25:38.905123 2665 leader_election.cc:341] T 3c5ec23c9759493aa1c33d7f0ea7e904 P 75595c6cc3d54995ade109df7bbe8820 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 87604ec3fe8c43b688447d96bd4bfd3e (127.9.176.194:35133): Not found: Tablet not found: 3c5ec23c9759493aa1c33d7f0ea7e904
I20260808 22:25:38.905238 2665 leader_election.cc:302] T 3c5ec23c9759493aa1c33d7f0ea7e904 P 75595c6cc3d54995ade109df7bbe8820 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 75595c6cc3d54995ade109df7bbe8820; no voters: 87604ec3fe8c43b688447d96bd4bfd3e, a606b1ce23654e70be927e2072afa5c6
I20260808 22:25:38.905378 3273 raft_consensus.cc:2726] T 3c5ec23c9759493aa1c33d7f0ea7e904 P 75595c6cc3d54995ade109df7bbe8820 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 22:25:39.493357 3275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:39.492982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:39769 (local address 127.9.176.193:32811)
0808 22:25:39.493206 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:39.493261 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:39769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":73,"threads_started":1}
I20260808 22:25:39.660429 3276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:39.660066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:39769 (local address 127.9.176.194:58265)
0808 22:25:39.660288 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:39.660339 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:39769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":69,"threads_started":1}
I20260808 22:25:39.836177 3278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:39.835745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:39769 (local address 127.9.176.195:50125)
0808 22:25:39.836023 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:39.836082 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:39769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":72,"threads_started":1}
I20260808 22:25:40.494455 3279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:40.494021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:39769 (local address 127.9.176.193:39703)
0808 22:25:40.494301 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:40.494359 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:39769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":76,"threads_started":1}
I20260808 22:25:40.661705 3280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:40.661224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:39769 (local address 127.9.176.194:59605)
0808 22:25:40.661520 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:40.661588 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:39769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":81,"threads_started":1}
I20260808 22:25:40.837286 3281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:40.836862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:39769 (local address 127.9.176.195:55409)
0808 22:25:40.837136 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:40.837189 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:39769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":109,"threads_started":1}
I20260808 22:25:41.495640 3285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:41.495203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:39769 (local address 127.9.176.193:45599)
0808 22:25:41.495473 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:41.495533 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:39769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":88,"threads_started":1}
I20260808 22:25:41.662858 3286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:41.662437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:39769 (local address 127.9.176.194:42113)
0808 22:25:41.662704 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:41.662761 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:39769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":89,"threads_started":1}
I20260808 22:25:41.838320 3287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:41.837923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:39769 (local address 127.9.176.195:35875)
0808 22:25:41.838178 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:41.838230 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:39769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":88,"threads_started":1}
I20260808 22:25:42.496729 3288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:42.496298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:39769 (local address 127.9.176.193:51177)
0808 22:25:42.496573 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:42.496631 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:39769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":102,"threads_started":1}
I20260808 22:25:42.574455 3293 raft_consensus.cc:482] T 3c5ec23c9759493aa1c33d7f0ea7e904 P 75595c6cc3d54995ade109df7bbe8820 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 22:25:42.574586 3293 raft_consensus.cc:504] T 3c5ec23c9759493aa1c33d7f0ea7e904 P 75595c6cc3d54995ade109df7bbe8820 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87604ec3fe8c43b688447d96bd4bfd3e" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 35133 } } peers { permanent_uuid: "a606b1ce23654e70be927e2072afa5c6" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 35701 } } peers { permanent_uuid: "75595c6cc3d54995ade109df7bbe8820" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 34743 } }
I20260808 22:25:42.574957 3293 leader_election.cc:288] T 3c5ec23c9759493aa1c33d7f0ea7e904 P 75595c6cc3d54995ade109df7bbe8820 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 87604ec3fe8c43b688447d96bd4bfd3e (127.9.176.194:35133), a606b1ce23654e70be927e2072afa5c6 (127.9.176.195:35701)
W20260808 22:25:42.575703 2665 leader_election.cc:341] T 3c5ec23c9759493aa1c33d7f0ea7e904 P 75595c6cc3d54995ade109df7bbe8820 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 87604ec3fe8c43b688447d96bd4bfd3e (127.9.176.194:35133): Not found: Tablet not found: 3c5ec23c9759493aa1c33d7f0ea7e904
W20260808 22:25:42.575820 2662 leader_election.cc:341] T 3c5ec23c9759493aa1c33d7f0ea7e904 P 75595c6cc3d54995ade109df7bbe8820 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a606b1ce23654e70be927e2072afa5c6 (127.9.176.195:35701): Not found: Tablet not found: 3c5ec23c9759493aa1c33d7f0ea7e904
I20260808 22:25:42.575887 2662 leader_election.cc:302] T 3c5ec23c9759493aa1c33d7f0ea7e904 P 75595c6cc3d54995ade109df7bbe8820 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 75595c6cc3d54995ade109df7bbe8820; no voters: 87604ec3fe8c43b688447d96bd4bfd3e, a606b1ce23654e70be927e2072afa5c6
I20260808 22:25:42.576018 3293 raft_consensus.cc:2726] T 3c5ec23c9759493aa1c33d7f0ea7e904 P 75595c6cc3d54995ade109df7bbe8820 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 22:25:42.663861 3294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:42.663479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:39769 (local address 127.9.176.194:32985)
0808 22:25:42.663722 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:42.663773 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:39769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":73,"threads_started":1}
I20260808 22:25:42.839349 3295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:42.838940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:39769 (local address 127.9.176.195:40801)
0808 22:25:42.839203 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:42.839259 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:39769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":82,"threads_started":1}
I20260808 22:25:43.497923 3297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:43.497503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:39769 (local address 127.9.176.193:40025)
0808 22:25:43.497766 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:43.497823 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:39769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":76,"threads_started":1}
I20260808 22:25:43.664940 3301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:43.664483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:39769 (local address 127.9.176.194:52427)
0808 22:25:43.664759 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:43.664816 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:39769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":112,"threads_started":1}
I20260808 22:25:43.840476 3302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:43.840059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:39769 (local address 127.9.176.195:58819)
0808 22:25:43.840317 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:43.840377 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:39769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":79,"threads_started":1}
I20260808 22:25:44.499219 3303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:44.498822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:39769 (local address 127.9.176.193:58171)
0808 22:25:44.499070 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:44.499125 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:39769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":73,"threads_started":1}
I20260808 22:25:44.666302 3304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:44.665837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:39769 (local address 127.9.176.194:55703)
0808 22:25:44.666120 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:44.666187 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:39769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":87,"threads_started":1}
I20260808 22:25:44.830566 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 2581
I20260808 22:25:44.841929 3306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:44.841494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:39769 (local address 127.9.176.195:49393)
0808 22:25:44.841764 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:44.841828 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.254:39769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":82,"threads_started":1}
I20260808 22:25:44.842859 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 2839
I20260808 22:25:44.852497 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 3043
2026-08-08T22:25:44Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59344 ms)
[----------] 30 tests from SecurityITest (238546 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 22:25:44 dist-test-slave-ncp9 krb5kdc[3309](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 22:25:44 dist-test-slave-ncp9 krb5kdc[3309](info): set up 2 sockets
Aug 08 22:25:44 dist-test-slave-ncp9 krb5kdc[3309](info): commencing operation
krb5kdc: starting...
W20260808 22:25:45.984230 9923 mini_kdc.cc:121] Time spent starting KDC: real 1.103s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/krb5kdc/test-user.keytab.
Aug 08 22:25:46 dist-test-slave-ncp9 krb5kdc[3309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227946, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T22:25:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:25:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.176.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.176.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.176.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab.
Entry for principal kudu/127.9.176.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab.
I20260808 22:25:46.101946 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:41841
--webserver_interface=127.9.176.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab
--principal=kudu/127.9.176.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:43299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.176.254:41841
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:25:46.209993 3330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:25:46.210264 3330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:25:46.210317 3330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:25:46.214186 3330 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:25:46.214239 3330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:25:46.214264 3330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:25:46.214284 3330 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:25:46.214304 3330 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:25:46.219327 3330 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:43299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.176.254:41841
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab
--principal=kudu/127.9.176.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:41841
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.176.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3330
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:25:46.220310 3330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:25:46.221338 3330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:25:46.229873 3339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:25:46.229982 3340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:25:46.230036 3330 server_base.cc:1034] running on GCE node
W20260808 22:25:46.229903 3342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:25:46.230724 3330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:25:46.231858 3330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:25:46.233017 3330 hybrid_clock.cc:648] HybridClock initialized: now 1786227946232992 us; error 39 us; skew 500 ppm
Aug 08 22:25:46 dist-test-slave-ncp9 krb5kdc[3309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227946, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:25:46.235746 3330 init.cc:377] Logged in from keytab as kudu/127.9.176.254@KRBTEST.COM (short username kudu)
I20260808 22:25:46.236650 3330 webserver.cc:460] Webserver started at http://127.9.176.254:37987/ using document root <none> and password file <none>
I20260808 22:25:46.237228 3330 fs_manager.cc:359] Metadata directory not provided
I20260808 22:25:46.237291 3330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:25:46.237506 3330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:25:46.239526 3330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/minicluster-data/master-0/data/instance:
uuid: "21e1c0d6a2b141b89bb4ffbdd8bc410d"
format_stamp: "Formatted at 2026-08-08 22:25:46 on dist-test-slave-ncp9"
server_key: "5e0b900ec422d468917ebd0271703b00"
server_key_iv: "f863aa33bc9e91f6fb6cb101e3933f83"
server_key_version: "encryptionkey@0"
I20260808 22:25:46.240074 3330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/minicluster-data/master-0/wal/instance:
uuid: "21e1c0d6a2b141b89bb4ffbdd8bc410d"
format_stamp: "Formatted at 2026-08-08 22:25:46 on dist-test-slave-ncp9"
server_key: "5e0b900ec422d468917ebd0271703b00"
server_key_iv: "f863aa33bc9e91f6fb6cb101e3933f83"
server_key_version: "encryptionkey@0"
I20260808 22:25:46.243712 3330 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260808 22:25:46.246042 3350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:25:46.247218 3330 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 22:25:46.247330 3330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/minicluster-data/master-0/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/minicluster-data/master-0/wal
uuid: "21e1c0d6a2b141b89bb4ffbdd8bc410d"
format_stamp: "Formatted at 2026-08-08 22:25:46 on dist-test-slave-ncp9"
server_key: "5e0b900ec422d468917ebd0271703b00"
server_key_iv: "f863aa33bc9e91f6fb6cb101e3933f83"
server_key_version: "encryptionkey@0"
I20260808 22:25:46.247421 3330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:25:46.270973 3330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:25:46.273667 3330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:25:46.273834 3330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:25:46.280560 3330 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.254:41841
I20260808 22:25:46.280639 3431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.254:41841 every 8 connection(s)
I20260808 22:25:46.281574 3330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
I20260808 22:25:46.284312 3432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:25:46.287875 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 3330
I20260808 22:25:46.287992 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/minicluster-data/master-0/wal/instance
I20260808 22:25:46.288255 9923 external_mini_cluster.cc:1428] Setting key 7421ba24ee08fe42bb5497285b5a112a
I20260808 22:25:46.290035 3432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21e1c0d6a2b141b89bb4ffbdd8bc410d: Bootstrap starting.
I20260808 22:25:46.292651 3432 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 21e1c0d6a2b141b89bb4ffbdd8bc410d: Neither blocks nor log segments found. Creating new log.
I20260808 22:25:46.293532 3432 log.cc:824] T 00000000000000000000000000000000 P 21e1c0d6a2b141b89bb4ffbdd8bc410d: Log is configured to *not* fsync() on all Append() calls
I20260808 22:25:46.295337 3432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21e1c0d6a2b141b89bb4ffbdd8bc410d: No bootstrap required, opened a new log
Aug 08 22:25:46 dist-test-slave-ncp9 krb5kdc[3309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227946, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
I20260808 22:25:46.298895 3432 raft_consensus.cc:348] T 00000000000000000000000000000000 P 21e1c0d6a2b141b89bb4ffbdd8bc410d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21e1c0d6a2b141b89bb4ffbdd8bc410d" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 41841 } }
I20260808 22:25:46.299129 3432 raft_consensus.cc:374] T 00000000000000000000000000000000 P 21e1c0d6a2b141b89bb4ffbdd8bc410d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:25:46.299186 3432 raft_consensus.cc:729] T 00000000000000000000000000000000 P 21e1c0d6a2b141b89bb4ffbdd8bc410d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21e1c0d6a2b141b89bb4ffbdd8bc410d, State: Initialized, Role: FOLLOWER
I20260808 22:25:46.299597 3432 consensus_queue.cc:260] T 00000000000000000000000000000000 P 21e1c0d6a2b141b89bb4ffbdd8bc410d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21e1c0d6a2b141b89bb4ffbdd8bc410d" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 41841 } }
I20260808 22:25:46.299726 3432 raft_consensus.cc:388] T 00000000000000000000000000000000 P 21e1c0d6a2b141b89bb4ffbdd8bc410d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:25:46.299791 3432 raft_consensus.cc:482] T 00000000000000000000000000000000 P 21e1c0d6a2b141b89bb4ffbdd8bc410d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:25:46.299885 3432 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 21e1c0d6a2b141b89bb4ffbdd8bc410d [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:25:46.300673 3432 raft_consensus.cc:504] T 00000000000000000000000000000000 P 21e1c0d6a2b141b89bb4ffbdd8bc410d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21e1c0d6a2b141b89bb4ffbdd8bc410d" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 41841 } }
I20260808 22:25:46.301038 3432 leader_election.cc:302] T 00000000000000000000000000000000 P 21e1c0d6a2b141b89bb4ffbdd8bc410d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 21e1c0d6a2b141b89bb4ffbdd8bc410d; no voters:
I20260808 22:25:46.301357 3432 leader_election.cc:288] T 00000000000000000000000000000000 P 21e1c0d6a2b141b89bb4ffbdd8bc410d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:25:46.301463 3437 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 21e1c0d6a2b141b89bb4ffbdd8bc410d [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:25:46.301656 3437 raft_consensus.cc:686] T 00000000000000000000000000000000 P 21e1c0d6a2b141b89bb4ffbdd8bc410d [term 1 LEADER]: Becoming Leader. State: Replica: 21e1c0d6a2b141b89bb4ffbdd8bc410d, State: Running, Role: LEADER
I20260808 22:25:46.302014 3437 consensus_queue.cc:237] T 00000000000000000000000000000000 P 21e1c0d6a2b141b89bb4ffbdd8bc410d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21e1c0d6a2b141b89bb4ffbdd8bc410d" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 41841 } }
I20260808 22:25:46.302417 3435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:46.289617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38912 (local address 127.9.176.254:41841)
0808 22:25:46.290025 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0808 22:25:46.290046 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0808 22:25:46.290089 (+ 43us) server_negotiation.cc:407] Connection header received
0808 22:25:46.290882 (+ 793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:25:46.290904 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:25:46.291315 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:25:46.291687 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:25:46.292350 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:46.293864 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:46.294735 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:46.295201 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:46.295343 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:46.297718 (+ 2375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:25:46.297723 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:25:46.297736 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:25:46.297762 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:25:46.299402 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:25:46.299866 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:25:46.299872 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:25:46.299879 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:25:46.300013 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:25:46.300271 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:25:46.300275 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:25:46.300277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:25:46.300680 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:25:46.300854 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0808 22:25:46.301152 (+ 298us) server_negotiation.cc:299] Negotiation successful
0808 22:25:46.301449 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":108,"threads_started":1}
I20260808 22:25:46.302568 3432 sys_catalog.cc:564] T 00000000000000000000000000000000 P 21e1c0d6a2b141b89bb4ffbdd8bc410d [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:25:46.305140 3439 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21e1c0d6a2b141b89bb4ffbdd8bc410d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 21e1c0d6a2b141b89bb4ffbdd8bc410d. Latest consensus state: current_term: 1 leader_uuid: "21e1c0d6a2b141b89bb4ffbdd8bc410d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21e1c0d6a2b141b89bb4ffbdd8bc410d" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 41841 } } }
I20260808 22:25:46.305256 3439 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21e1c0d6a2b141b89bb4ffbdd8bc410d [sys.catalog]: This master's current role is: LEADER
I20260808 22:25:46.305516 3438 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21e1c0d6a2b141b89bb4ffbdd8bc410d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "21e1c0d6a2b141b89bb4ffbdd8bc410d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21e1c0d6a2b141b89bb4ffbdd8bc410d" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 41841 } } }
I20260808 22:25:46.305626 3438 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21e1c0d6a2b141b89bb4ffbdd8bc410d [sys.catalog]: This master's current role is: LEADER
W20260808 22:25:46.307364 3457 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 21e1c0d6a2b141b89bb4ffbdd8bc410d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 22:25:46.307436 3457 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 22:25:46.307569 3458 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:25:46.308454 3458 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:25:46.313663 3458 catalog_manager.cc:1347] Generated new cluster ID: 2cbcb38a1e224f59ad40d12b64ab5f01
I20260808 22:25:46.313730 3458 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:25:46.325835 3458 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:25:46.327020 3458 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:25:46.341459 3458 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 21e1c0d6a2b141b89bb4ffbdd8bc410d: Generated new TSK 0
I20260808 22:25:46.342286 3458 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.176.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.176.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.176.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.193.keytab.
Entry for principal kudu/127.9.176.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.193.keytab.
I20260808 22:25:46.378916 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.193:0
--local_ip_for_outbound_sockets=127.9.176.193
--webserver_interface=127.9.176.193
--webserver_port=0
--tserver_master_addrs=127.9.176.254:41841
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.193.keytab
--principal=kudu/127.9.176.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:43299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:25:46.488678 3464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:25:46.488937 3464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:25:46.488981 3464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:25:46.492810 3464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:25:46.492858 3464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:25:46.492928 3464 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.193
I20260808 22:25:46.497984 3464 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:43299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.193.keytab
--principal=kudu/127.9.176.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.176.193
--webserver_port=0
--tserver_master_addrs=127.9.176.254:41841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3464
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.193
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:25:46.498970 3464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:25:46.499930 3464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:25:46.509509 3472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:25:46.509505 3470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:25:46.509507 3479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:25:46.509562 3464 server_base.cc:1034] running on GCE node
I20260808 22:25:46.510203 3464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:25:46.510802 3464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:25:46.511976 3464 hybrid_clock.cc:648] HybridClock initialized: now 1786227946511949 us; error 27 us; skew 500 ppm
Aug 08 22:25:46 dist-test-slave-ncp9 krb5kdc[3309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227946, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:25:46.514710 3464 init.cc:377] Logged in from keytab as kudu/127.9.176.193@KRBTEST.COM (short username kudu)
I20260808 22:25:46.515555 3464 webserver.cc:460] Webserver started at http://127.9.176.193:38193/ using document root <none> and password file <none>
I20260808 22:25:46.516093 3464 fs_manager.cc:359] Metadata directory not provided
I20260808 22:25:46.516158 3464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:25:46.516359 3464 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:25:46.518393 3464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/minicluster-data/ts-0/data/instance:
uuid: "eb0243a710734e6f9989e10dac4f70aa"
format_stamp: "Formatted at 2026-08-08 22:25:46 on dist-test-slave-ncp9"
server_key: "e7f1987f1ec1a185881911e5e2269f0e"
server_key_iv: "b65a826fa119b2ab47924df661609ddc"
server_key_version: "encryptionkey@0"
I20260808 22:25:46.518940 3464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/minicluster-data/ts-0/wal/instance:
uuid: "eb0243a710734e6f9989e10dac4f70aa"
format_stamp: "Formatted at 2026-08-08 22:25:46 on dist-test-slave-ncp9"
server_key: "e7f1987f1ec1a185881911e5e2269f0e"
server_key_iv: "b65a826fa119b2ab47924df661609ddc"
server_key_version: "encryptionkey@0"
I20260808 22:25:46.522506 3464 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 22:25:46.524736 3492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:25:46.526038 3464 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 22:25:46.526135 3464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/minicluster-data/ts-0/wal
uuid: "eb0243a710734e6f9989e10dac4f70aa"
format_stamp: "Formatted at 2026-08-08 22:25:46 on dist-test-slave-ncp9"
server_key: "e7f1987f1ec1a185881911e5e2269f0e"
server_key_iv: "b65a826fa119b2ab47924df661609ddc"
server_key_version: "encryptionkey@0"
I20260808 22:25:46.526232 3464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:25:46.548674 3464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:25:46.551800 3464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:25:46.551978 3464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:25:46.552580 3464 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:25:46.553463 3464 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:25:46.553511 3464 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:25:46.553553 3464 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:25:46.553582 3464 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:25:46.562211 3464 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.193:41243
I20260808 22:25:46.562240 3656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.193:41243 every 8 connection(s)
I20260808 22:25:46.563086 3464 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
I20260808 22:25:46.565137 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 3464
I20260808 22:25:46.565254 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/minicluster-data/ts-0/wal/instance
I20260808 22:25:46.565532 9923 external_mini_cluster.cc:1428] Setting key cddbb25534eb8bafa2333bcfc80cb524
Aug 08 22:25:46 dist-test-slave-ncp9 krb5kdc[3309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227946, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.193@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.176.194@KRBTEST.COM; defaulting to no policy
I20260808 22:25:46.577790 3435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:46.564925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.193:59207 (local address 127.9.176.254:41841)
0808 22:25:46.565067 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0808 22:25:46.565071 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:25:46.565945 (+ 874us) server_negotiation.cc:407] Connection header received
0808 22:25:46.567313 (+ 1368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:25:46.567317 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:25:46.567367 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:25:46.567445 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:25:46.568604 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:46.569346 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:46.570503 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:46.570753 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:46.570842 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:46.573543 (+ 2701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:25:46.573546 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:25:46.573548 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:25:46.573572 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:25:46.575941 (+ 2369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:25:46.576422 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:25:46.576425 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:25:46.576427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:25:46.576517 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:25:46.576805 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:25:46.576807 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:25:46.576808 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:25:46.577010 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:25:46.577085 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0808 22:25:46.577558 (+ 473us) server_negotiation.cc:299] Negotiation successful
0808 22:25:46.577667 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.176.194@KRBTEST.COM" created.
I20260808 22:25:46.578748 3665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:46.565308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:41841 (local address 127.9.176.193:59207)
0808 22:25:46.565782 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:46.565834 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 22:25:46.567106 (+ 1272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:25:46.567602 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:25:46.567608 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:25:46.567955 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:25:46.568450 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:46.568463 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:46.569475 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:46.569478 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:46.570383 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:46.570393 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:46.570893 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:46.570897 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:46.571065 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:46.571638 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:25:46.571654 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:25:46.573379 (+ 1725us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:25:46.576048 (+ 2669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:25:46.576053 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:25:46.576065 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:25:46.576318 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:25:46.576590 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:25:46.576592 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:25:46.576594 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:25:46.576737 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:25:46.577087 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:25:46.577092 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:25:46.577331 (+ 239us) client_negotiation.cc:769] Sending connection context
0808 22:25:46.577509 (+ 178us) client_negotiation.cc:240] Negotiation successful
0808 22:25:46.577791 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":105,"threads_started":1}
I20260808 22:25:46.580721 3662 heartbeater.cc:347] Connected to a master server at 127.9.176.254:41841
I20260808 22:25:46.581000 3662 heartbeater.cc:464] Registering TS with master...
I20260808 22:25:46.581588 3662 heartbeater.cc:511] Master 127.9.176.254:41841 requested a full tablet report, sending...
I20260808 22:25:46.583359 3375 ts_manager.cc:194] Registered new tserver with Master: eb0243a710734e6f9989e10dac4f70aa (127.9.176.193:41243)
I20260808 22:25:46.584947 3375 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.176.193@KRBTEST.COM'} at 127.9.176.193:59207
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.176.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.194.keytab.
Entry for principal kudu/127.9.176.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.194.keytab.
I20260808 22:25:46.592998 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.194:0
--local_ip_for_outbound_sockets=127.9.176.194
--webserver_interface=127.9.176.194
--webserver_port=0
--tserver_master_addrs=127.9.176.254:41841
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.194.keytab
--principal=kudu/127.9.176.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:43299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:25:46.701857 3669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:25:46.702132 3669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:25:46.702214 3669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:25:46.706517 3669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:25:46.706573 3669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:25:46.706634 3669 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.194
I20260808 22:25:46.711776 3669 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:43299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.194.keytab
--principal=kudu/127.9.176.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.176.194
--webserver_port=0
--tserver_master_addrs=127.9.176.254:41841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3669
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.194
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:25:46.712747 3669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:25:46.713740 3669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:25:46.722958 3678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:25:46.723002 3681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:25:46.723136 3669 server_base.cc:1034] running on GCE node
W20260808 22:25:46.722985 3679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:25:46.723606 3669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:25:46.724150 3669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:25:46.725319 3669 hybrid_clock.cc:648] HybridClock initialized: now 1786227946725303 us; error 23 us; skew 500 ppm
Aug 08 22:25:46 dist-test-slave-ncp9 krb5kdc[3309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227946, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:25:46.728628 3669 init.cc:377] Logged in from keytab as kudu/127.9.176.194@KRBTEST.COM (short username kudu)
I20260808 22:25:46.729485 3669 webserver.cc:460] Webserver started at http://127.9.176.194:38779/ using document root <none> and password file <none>
I20260808 22:25:46.730018 3669 fs_manager.cc:359] Metadata directory not provided
I20260808 22:25:46.730082 3669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:25:46.730291 3669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:25:46.732215 3669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/minicluster-data/ts-1/data/instance:
uuid: "b797dba267504682bfff47ea3a4d7a53"
format_stamp: "Formatted at 2026-08-08 22:25:46 on dist-test-slave-ncp9"
server_key: "4981fd504ed7b8fc7dbd02812178fc94"
server_key_iv: "4fdd1e4af8c4c2915734587cbec84f20"
server_key_version: "encryptionkey@0"
I20260808 22:25:46.732744 3669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/minicluster-data/ts-1/wal/instance:
uuid: "b797dba267504682bfff47ea3a4d7a53"
format_stamp: "Formatted at 2026-08-08 22:25:46 on dist-test-slave-ncp9"
server_key: "4981fd504ed7b8fc7dbd02812178fc94"
server_key_iv: "4fdd1e4af8c4c2915734587cbec84f20"
server_key_version: "encryptionkey@0"
I20260808 22:25:46.736213 3669 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260808 22:25:46.738467 3691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:25:46.739575 3669 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 22:25:46.739670 3669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/minicluster-data/ts-1/wal
uuid: "b797dba267504682bfff47ea3a4d7a53"
format_stamp: "Formatted at 2026-08-08 22:25:46 on dist-test-slave-ncp9"
server_key: "4981fd504ed7b8fc7dbd02812178fc94"
server_key_iv: "4fdd1e4af8c4c2915734587cbec84f20"
server_key_version: "encryptionkey@0"
I20260808 22:25:46.739766 3669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:25:46.752372 3669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:25:46.754877 3669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:25:46.755038 3669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:25:46.755563 3669 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:25:46.756330 3669 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:25:46.756374 3669 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:25:46.756417 3669 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:25:46.756444 3669 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:25:46.764662 3669 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.194:44283
I20260808 22:25:46.764686 3854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.194:44283 every 8 connection(s)
I20260808 22:25:46.765511 3669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
I20260808 22:25:46.768931 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 3669
I20260808 22:25:46.769065 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/minicluster-data/ts-1/wal/instance
I20260808 22:25:46.769363 9923 external_mini_cluster.cc:1428] Setting key 63abd77a64fd92d6579728ab0b52d6be
Aug 08 22:25:46 dist-test-slave-ncp9 krb5kdc[3309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227946, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.194@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.176.195@KRBTEST.COM; defaulting to no policy
I20260808 22:25:46.778712 3435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:46.767143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.194:56159 (local address 127.9.176.254:41841)
0808 22:25:46.767274 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0808 22:25:46.767280 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 22:25:46.768115 (+ 835us) server_negotiation.cc:407] Connection header received
0808 22:25:46.769535 (+ 1420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:25:46.769539 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:25:46.769597 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:25:46.769687 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:25:46.770923 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:46.771666 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:46.772801 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:46.773090 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:46.773181 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:46.775441 (+ 2260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:25:46.775445 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:25:46.775447 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:25:46.775471 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:25:46.776827 (+ 1356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:25:46.777310 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:25:46.777313 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:25:46.777314 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:25:46.777390 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:25:46.777710 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:25:46.777714 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:25:46.777715 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:25:46.777888 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:25:46.777973 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0808 22:25:46.778435 (+ 462us) server_negotiation.cc:299] Negotiation successful
0808 22:25:46.778556 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260808 22:25:46.779433 3858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:46.767524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:41841 (local address 127.9.176.194:56159)
0808 22:25:46.767970 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:46.768011 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 22:25:46.769296 (+ 1285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:25:46.769862 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:25:46.769870 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:25:46.770239 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:25:46.770788 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:46.770806 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:46.771768 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:46.771771 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:46.772688 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:46.772698 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:46.773232 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:46.773236 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:46.773391 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:46.773904 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:25:46.773921 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:25:46.775290 (+ 1369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:25:46.776950 (+ 1660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:25:46.776956 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:25:46.776970 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:25:46.777216 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:25:46.777472 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:25:46.777476 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:25:46.777478 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:25:46.777615 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:25:46.777989 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:25:46.777995 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:25:46.778215 (+ 220us) client_negotiation.cc:769] Sending connection context
0808 22:25:46.778373 (+ 158us) client_negotiation.cc:240] Negotiation successful
0808 22:25:46.778680 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":114,"threads_started":1}
I20260808 22:25:46.780579 3855 heartbeater.cc:347] Connected to a master server at 127.9.176.254:41841
I20260808 22:25:46.780790 3855 heartbeater.cc:464] Registering TS with master...
I20260808 22:25:46.781265 3855 heartbeater.cc:511] Master 127.9.176.254:41841 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.176.195@KRBTEST.COM" created.
I20260808 22:25:46.782330 3375 ts_manager.cc:194] Registered new tserver with Master: b797dba267504682bfff47ea3a4d7a53 (127.9.176.194:44283)
I20260808 22:25:46.783025 3375 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.176.194@KRBTEST.COM'} at 127.9.176.194:56159
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.176.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.195.keytab.
Entry for principal kudu/127.9.176.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.195.keytab.
I20260808 22:25:46.796085 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.195:0
--local_ip_for_outbound_sockets=127.9.176.195
--webserver_interface=127.9.176.195
--webserver_port=0
--tserver_master_addrs=127.9.176.254:41841
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.195.keytab
--principal=kudu/127.9.176.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:43299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:25:46.902822 3863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:25:46.903053 3863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:25:46.903098 3863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:25:46.907153 3863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:25:46.907203 3863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:25:46.907262 3863 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.195
I20260808 22:25:46.912246 3863 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:43299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.195.keytab
--principal=kudu/127.9.176.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.176.195
--webserver_port=0
--tserver_master_addrs=127.9.176.254:41841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3863
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.195
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:25:46.913188 3863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:25:46.914129 3863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:25:46.923074 3871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:25:46.923146 3870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:25:46.923122 3876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:25:46.923237 3863 server_base.cc:1034] running on GCE node
I20260808 22:25:46.923727 3863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:25:46.924331 3863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:25:46.925498 3863 hybrid_clock.cc:648] HybridClock initialized: now 1786227946925484 us; error 27 us; skew 500 ppm
Aug 08 22:25:46 dist-test-slave-ncp9 krb5kdc[3309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227946, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:25:46.928179 3863 init.cc:377] Logged in from keytab as kudu/127.9.176.195@KRBTEST.COM (short username kudu)
I20260808 22:25:46.928977 3863 webserver.cc:460] Webserver started at http://127.9.176.195:38103/ using document root <none> and password file <none>
I20260808 22:25:46.929533 3863 fs_manager.cc:359] Metadata directory not provided
I20260808 22:25:46.929601 3863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:25:46.929803 3863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:25:46.931732 3863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/minicluster-data/ts-2/data/instance:
uuid: "4317cf1cf08843ce886bdc9e7500c64e"
format_stamp: "Formatted at 2026-08-08 22:25:46 on dist-test-slave-ncp9"
server_key: "a6c86d22103101b0e94596e69b4c4b6c"
server_key_iv: "7d8a7329ab093cd069d781d081cecfdb"
server_key_version: "encryptionkey@0"
I20260808 22:25:46.932263 3863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/minicluster-data/ts-2/wal/instance:
uuid: "4317cf1cf08843ce886bdc9e7500c64e"
format_stamp: "Formatted at 2026-08-08 22:25:46 on dist-test-slave-ncp9"
server_key: "a6c86d22103101b0e94596e69b4c4b6c"
server_key_iv: "7d8a7329ab093cd069d781d081cecfdb"
server_key_version: "encryptionkey@0"
I20260808 22:25:46.935647 3863 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260808 22:25:46.937876 3884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:25:46.938947 3863 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 22:25:46.939051 3863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/minicluster-data/ts-2/wal
uuid: "4317cf1cf08843ce886bdc9e7500c64e"
format_stamp: "Formatted at 2026-08-08 22:25:46 on dist-test-slave-ncp9"
server_key: "a6c86d22103101b0e94596e69b4c4b6c"
server_key_iv: "7d8a7329ab093cd069d781d081cecfdb"
server_key_version: "encryptionkey@0"
I20260808 22:25:46.939149 3863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:25:46.959407 3863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:25:46.962445 3863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:25:46.962613 3863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:25:46.963169 3863 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:25:46.963937 3863 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:25:46.963984 3863 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:25:46.964033 3863 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:25:46.964061 3863 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:25:46.972299 3863 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.195:37195
I20260808 22:25:46.972321 4053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.195:37195 every 8 connection(s)
I20260808 22:25:46.973140 3863 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
I20260808 22:25:46.981415 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 3863
I20260808 22:25:46.981547 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/minicluster-data/ts-2/wal/instance
Aug 08 22:25:46 dist-test-slave-ncp9 krb5kdc[3309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227946, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.195@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
I20260808 22:25:46.981817 9923 external_mini_cluster.cc:1428] Setting key 8ce247083a1b2b9ac36fbcccb1666146
I20260808 22:25:46.985808 3435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:46.974736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.195:60153 (local address 127.9.176.254:41841)
0808 22:25:46.974862 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0808 22:25:46.974866 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:25:46.975605 (+ 739us) server_negotiation.cc:407] Connection header received
0808 22:25:46.976920 (+ 1315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:25:46.976923 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:25:46.976983 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:25:46.977055 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:25:46.978140 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:46.978671 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:46.979770 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:46.979989 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:46.980075 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:46.982523 (+ 2448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:25:46.982527 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:25:46.982528 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:25:46.982557 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:25:46.984042 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:25:46.984474 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:25:46.984476 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:25:46.984478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:25:46.984552 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:25:46.984800 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:25:46.984803 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:25:46.984804 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:25:46.985024 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:25:46.985099 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0808 22:25:46.985599 (+ 500us) server_negotiation.cc:299] Negotiation successful
0808 22:25:46.985696 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260808 22:25:46.986591 4057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:46.975052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:41841 (local address 127.9.176.195:60153)
0808 22:25:46.975468 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:46.975508 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 22:25:46.976690 (+ 1182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:25:46.977230 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:25:46.977237 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:25:46.977555 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:25:46.977995 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:46.978008 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:46.978788 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:46.978790 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:46.979646 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:46.979656 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:46.980121 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:46.980135 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:46.980290 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:46.980729 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:25:46.980745 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:25:46.982231 (+ 1486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:25:46.984149 (+ 1918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:25:46.984153 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:25:46.984162 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:25:46.984392 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:25:46.984618 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:25:46.984620 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:25:46.984622 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:25:46.984738 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:25:46.985133 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:25:46.985139 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:25:46.985378 (+ 239us) client_negotiation.cc:769] Sending connection context
0808 22:25:46.985553 (+ 175us) client_negotiation.cc:240] Negotiation successful
0808 22:25:46.985808 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":112,"threads_started":1}
I20260808 22:25:46.987651 4055 heartbeater.cc:347] Connected to a master server at 127.9.176.254:41841
I20260808 22:25:46.987871 4055 heartbeater.cc:464] Registering TS with master...
I20260808 22:25:46.988327 4055 heartbeater.cc:511] Master 127.9.176.254:41841 requested a full tablet report, sending...
I20260808 22:25:46.989311 3375 ts_manager.cc:194] Registered new tserver with Master: 4317cf1cf08843ce886bdc9e7500c64e (127.9.176.195:37195)
I20260808 22:25:46.989876 3375 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.176.195@KRBTEST.COM'} at 127.9.176.195:60153
I20260808 22:25:46.995671 9923 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 22:25:47.004242 3435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:46.997484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38920 (local address 127.9.176.254:41841)
0808 22:25:46.997588 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0808 22:25:46.997592 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:25:46.997745 (+ 153us) server_negotiation.cc:407] Connection header received
0808 22:25:46.997867 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:25:46.997870 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:25:46.997912 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:25:46.997968 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:25:46.998610 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:46.999133 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:46.999946 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:47.000175 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:47.000260 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:47.001453 (+ 1193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:25:47.001457 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:25:47.001460 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:25:47.001486 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:25:47.002589 (+ 1103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:25:47.003031 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:25:47.003035 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:25:47.003036 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:25:47.003112 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:25:47.003381 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:25:47.003384 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:25:47.003386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:25:47.003586 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:25:47.003701 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0808 22:25:47.003979 (+ 278us) server_negotiation.cc:299] Negotiation successful
0808 22:25:47.004083 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260808 22:25:47.006630 3375 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38920:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 22:25:47.009521 3375 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 22:25:47.023283 4077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:47.017485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.195:37195 (local address 127.0.0.1:34056)
0808 22:25:47.018224 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:47.018238 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 22:25:47.018363 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:25:47.019191 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:25:47.019197 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:25:47.019231 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:25:47.019401 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:47.019409 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:47.020788 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:47.020791 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:47.021929 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:47.021937 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:47.022774 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:47.022778 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:47.022882 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:47.022917 (+ 35us) client_negotiation.cc:769] Sending connection context
0808 22:25:47.023020 (+ 103us) client_negotiation.cc:240] Negotiation successful
0808 22:25:47.023133 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":679,"mutex_wait_us":50,"spinlock_wait_cycles":3328,"thread_start_us":112,"threads_started":1}
I20260808 22:25:47.023317 4079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:47.018056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34056 (local address 127.9.176.195:37195)
0808 22:25:47.018732 (+ 676us) server_negotiation.cc:206] Beginning negotiation
0808 22:25:47.018746 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 22:25:47.018768 (+ 22us) server_negotiation.cc:407] Connection header received
0808 22:25:47.018842 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:25:47.018847 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:25:47.019038 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:25:47.019183 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:25:47.019534 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:47.020648 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:47.022034 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:47.022658 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:47.022764 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:47.022901 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0808 22:25:47.023076 (+ 175us) server_negotiation.cc:299] Negotiation successful
0808 22:25:47.023172 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":175,"threads_started":1}
I20260808 22:25:47.023638 4075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:47.017477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50392 (local address 127.9.176.193:41243)
0808 22:25:47.017911 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0808 22:25:47.017924 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 22:25:47.018282 (+ 358us) server_negotiation.cc:407] Connection header received
0808 22:25:47.018689 (+ 407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:25:47.018695 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:25:47.018880 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:25:47.019007 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:25:47.019487 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:47.020339 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:47.021868 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:47.022682 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:47.022773 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:47.022908 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0808 22:25:47.023404 (+ 496us) server_negotiation.cc:299] Negotiation successful
0808 22:25:47.023500 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":119,"threads_started":1}
I20260808 22:25:47.023311 4073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:47.017332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.193:41243 (local address 127.0.0.1:50392)
0808 22:25:47.018136 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:47.018184 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0808 22:25:47.018365 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:25:47.019203 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:25:47.019205 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:25:47.019232 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:25:47.019376 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:47.019380 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:47.020444 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:47.020448 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:47.021759 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:47.021770 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:47.022969 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:47.022972 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:47.023066 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:47.023087 (+ 21us) client_negotiation.cc:769] Sending connection context
0808 22:25:47.023157 (+ 70us) client_negotiation.cc:240] Negotiation successful
0808 22:25:47.023213 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"mutex_wait_us":25,"spinlock_wait_cycles":5120,"thread_start_us":74,"threads_started":1}
I20260808 22:25:47.023921 4074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:47.017987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.194:44283 (local address 127.0.0.1:50928)
0808 22:25:47.018598 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:47.018610 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 22:25:47.018687 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:25:47.019930 (+ 1243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:25:47.019933 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:25:47.019952 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:25:47.020029 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:47.020035 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:47.021272 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:47.021275 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:47.022396 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:47.022408 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:47.023583 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:47.023586 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:47.023650 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:47.023672 (+ 22us) client_negotiation.cc:769] Sending connection context
0808 22:25:47.023738 (+ 66us) client_negotiation.cc:240] Negotiation successful
0808 22:25:47.023803 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":94,"threads_started":1}
I20260808 22:25:47.024412 4081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:47.018283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50928 (local address 127.9.176.194:44283)
0808 22:25:47.019502 (+ 1219us) server_negotiation.cc:206] Beginning negotiation
0808 22:25:47.019517 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 22:25:47.019537 (+ 20us) server_negotiation.cc:407] Connection header received
0808 22:25:47.019608 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:25:47.019613 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:25:47.019791 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:25:47.019935 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:25:47.020120 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:47.021111 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:47.022534 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:47.023303 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:47.023403 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:47.023528 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0808 22:25:47.024180 (+ 652us) server_negotiation.cc:299] Negotiation successful
0808 22:25:47.024269 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1108,"thread_start_us":114,"threads_started":1}
I20260808 22:25:47.026422 3954 tablet_service.cc:1467] Processing CreateTablet for tablet ef2175d7105849e49961acc37682657f (DEFAULT_TABLE table=test-table [id=1fbfbed10aa3496fa07a4009c1cf5667]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:25:47.026475 3762 tablet_service.cc:1467] Processing CreateTablet for tablet ef2175d7105849e49961acc37682657f (DEFAULT_TABLE table=test-table [id=1fbfbed10aa3496fa07a4009c1cf5667]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:25:47.027387 3954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef2175d7105849e49961acc37682657f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:25:47.027387 3762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef2175d7105849e49961acc37682657f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:25:47.028743 3561 tablet_service.cc:1467] Processing CreateTablet for tablet ef2175d7105849e49961acc37682657f (DEFAULT_TABLE table=test-table [id=1fbfbed10aa3496fa07a4009c1cf5667]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:25:47.029739 3561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef2175d7105849e49961acc37682657f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:25:47.033774 4082 tablet_bootstrap.cc:492] T ef2175d7105849e49961acc37682657f P 4317cf1cf08843ce886bdc9e7500c64e: Bootstrap starting.
I20260808 22:25:47.035084 4083 tablet_bootstrap.cc:492] T ef2175d7105849e49961acc37682657f P b797dba267504682bfff47ea3a4d7a53: Bootstrap starting.
I20260808 22:25:47.035687 4082 tablet_bootstrap.cc:654] T ef2175d7105849e49961acc37682657f P 4317cf1cf08843ce886bdc9e7500c64e: Neither blocks nor log segments found. Creating new log.
I20260808 22:25:47.036233 4085 tablet_bootstrap.cc:492] T ef2175d7105849e49961acc37682657f P eb0243a710734e6f9989e10dac4f70aa: Bootstrap starting.
I20260808 22:25:47.036966 4082 log.cc:824] T ef2175d7105849e49961acc37682657f P 4317cf1cf08843ce886bdc9e7500c64e: Log is configured to *not* fsync() on all Append() calls
I20260808 22:25:47.037200 4083 tablet_bootstrap.cc:654] T ef2175d7105849e49961acc37682657f P b797dba267504682bfff47ea3a4d7a53: Neither blocks nor log segments found. Creating new log.
I20260808 22:25:47.038254 4085 tablet_bootstrap.cc:654] T ef2175d7105849e49961acc37682657f P eb0243a710734e6f9989e10dac4f70aa: Neither blocks nor log segments found. Creating new log.
I20260808 22:25:47.038296 4083 log.cc:824] T ef2175d7105849e49961acc37682657f P b797dba267504682bfff47ea3a4d7a53: Log is configured to *not* fsync() on all Append() calls
I20260808 22:25:47.039310 4082 tablet_bootstrap.cc:492] T ef2175d7105849e49961acc37682657f P 4317cf1cf08843ce886bdc9e7500c64e: No bootstrap required, opened a new log
I20260808 22:25:47.039467 4085 log.cc:824] T ef2175d7105849e49961acc37682657f P eb0243a710734e6f9989e10dac4f70aa: Log is configured to *not* fsync() on all Append() calls
I20260808 22:25:47.039505 4082 ts_tablet_manager.cc:1397] T ef2175d7105849e49961acc37682657f P 4317cf1cf08843ce886bdc9e7500c64e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 22:25:47.040040 4083 tablet_bootstrap.cc:492] T ef2175d7105849e49961acc37682657f P b797dba267504682bfff47ea3a4d7a53: No bootstrap required, opened a new log
I20260808 22:25:47.040234 4083 ts_tablet_manager.cc:1397] T ef2175d7105849e49961acc37682657f P b797dba267504682bfff47ea3a4d7a53: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260808 22:25:47.041344 4085 tablet_bootstrap.cc:492] T ef2175d7105849e49961acc37682657f P eb0243a710734e6f9989e10dac4f70aa: No bootstrap required, opened a new log
I20260808 22:25:47.041527 4085 ts_tablet_manager.cc:1397] T ef2175d7105849e49961acc37682657f P eb0243a710734e6f9989e10dac4f70aa: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.004s
I20260808 22:25:47.043128 4083 raft_consensus.cc:348] T ef2175d7105849e49961acc37682657f P b797dba267504682bfff47ea3a4d7a53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0243a710734e6f9989e10dac4f70aa" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 41243 } } peers { permanent_uuid: "4317cf1cf08843ce886bdc9e7500c64e" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37195 } } peers { permanent_uuid: "b797dba267504682bfff47ea3a4d7a53" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 44283 } }
I20260808 22:25:47.043383 4083 raft_consensus.cc:374] T ef2175d7105849e49961acc37682657f P b797dba267504682bfff47ea3a4d7a53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:25:47.043457 4083 raft_consensus.cc:729] T ef2175d7105849e49961acc37682657f P b797dba267504682bfff47ea3a4d7a53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b797dba267504682bfff47ea3a4d7a53, State: Initialized, Role: FOLLOWER
I20260808 22:25:47.043303 4082 raft_consensus.cc:348] T ef2175d7105849e49961acc37682657f P 4317cf1cf08843ce886bdc9e7500c64e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0243a710734e6f9989e10dac4f70aa" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 41243 } } peers { permanent_uuid: "4317cf1cf08843ce886bdc9e7500c64e" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37195 } } peers { permanent_uuid: "b797dba267504682bfff47ea3a4d7a53" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 44283 } }
I20260808 22:25:47.043612 4082 raft_consensus.cc:374] T ef2175d7105849e49961acc37682657f P 4317cf1cf08843ce886bdc9e7500c64e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:25:47.043670 4082 raft_consensus.cc:729] T ef2175d7105849e49961acc37682657f P 4317cf1cf08843ce886bdc9e7500c64e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4317cf1cf08843ce886bdc9e7500c64e, State: Initialized, Role: FOLLOWER
I20260808 22:25:47.043988 4083 consensus_queue.cc:260] T ef2175d7105849e49961acc37682657f P b797dba267504682bfff47ea3a4d7a53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0243a710734e6f9989e10dac4f70aa" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 41243 } } peers { permanent_uuid: "4317cf1cf08843ce886bdc9e7500c64e" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37195 } } peers { permanent_uuid: "b797dba267504682bfff47ea3a4d7a53" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 44283 } }
I20260808 22:25:47.044068 4082 consensus_queue.cc:260] T ef2175d7105849e49961acc37682657f P 4317cf1cf08843ce886bdc9e7500c64e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0243a710734e6f9989e10dac4f70aa" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 41243 } } peers { permanent_uuid: "4317cf1cf08843ce886bdc9e7500c64e" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37195 } } peers { permanent_uuid: "b797dba267504682bfff47ea3a4d7a53" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 44283 } }
I20260808 22:25:47.044718 4055 heartbeater.cc:503] Master 127.9.176.254:41841 was elected leader, sending a full tablet report...
I20260808 22:25:47.045035 4082 ts_tablet_manager.cc:1428] T ef2175d7105849e49961acc37682657f P 4317cf1cf08843ce886bdc9e7500c64e: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260808 22:25:47.045279 4085 raft_consensus.cc:348] T ef2175d7105849e49961acc37682657f P eb0243a710734e6f9989e10dac4f70aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0243a710734e6f9989e10dac4f70aa" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 41243 } } peers { permanent_uuid: "4317cf1cf08843ce886bdc9e7500c64e" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37195 } } peers { permanent_uuid: "b797dba267504682bfff47ea3a4d7a53" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 44283 } }
I20260808 22:25:47.045506 4085 raft_consensus.cc:374] T ef2175d7105849e49961acc37682657f P eb0243a710734e6f9989e10dac4f70aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:25:47.045589 4085 raft_consensus.cc:729] T ef2175d7105849e49961acc37682657f P eb0243a710734e6f9989e10dac4f70aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb0243a710734e6f9989e10dac4f70aa, State: Initialized, Role: FOLLOWER
I20260808 22:25:47.045981 4085 consensus_queue.cc:260] T ef2175d7105849e49961acc37682657f P eb0243a710734e6f9989e10dac4f70aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0243a710734e6f9989e10dac4f70aa" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 41243 } } peers { permanent_uuid: "4317cf1cf08843ce886bdc9e7500c64e" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37195 } } peers { permanent_uuid: "b797dba267504682bfff47ea3a4d7a53" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 44283 } }
I20260808 22:25:47.046082 4083 ts_tablet_manager.cc:1428] T ef2175d7105849e49961acc37682657f P b797dba267504682bfff47ea3a4d7a53: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260808 22:25:47.046111 3855 heartbeater.cc:503] Master 127.9.176.254:41841 was elected leader, sending a full tablet report...
I20260808 22:25:47.046553 3662 heartbeater.cc:503] Master 127.9.176.254:41841 was elected leader, sending a full tablet report...
I20260808 22:25:47.046876 4085 ts_tablet_manager.cc:1428] T ef2175d7105849e49961acc37682657f P eb0243a710734e6f9989e10dac4f70aa: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260808 22:25:47.060164 4092 raft_consensus.cc:482] T ef2175d7105849e49961acc37682657f P b797dba267504682bfff47ea3a4d7a53 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 22:25:47.060290 4092 raft_consensus.cc:504] T ef2175d7105849e49961acc37682657f P b797dba267504682bfff47ea3a4d7a53 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0243a710734e6f9989e10dac4f70aa" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 41243 } } peers { permanent_uuid: "4317cf1cf08843ce886bdc9e7500c64e" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37195 } } peers { permanent_uuid: "b797dba267504682bfff47ea3a4d7a53" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 44283 } }
I20260808 22:25:47.061230 4092 leader_election.cc:288] T ef2175d7105849e49961acc37682657f P b797dba267504682bfff47ea3a4d7a53 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb0243a710734e6f9989e10dac4f70aa (127.9.176.193:41243), 4317cf1cf08843ce886bdc9e7500c64e (127.9.176.195:37195)
W20260808 22:25:47.064174 3664 tablet.cc:2369] T ef2175d7105849e49961acc37682657f P eb0243a710734e6f9989e10dac4f70aa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 22:25:47.064885 4075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:47.061484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.194:60083 (local address 127.9.176.193:41243)
0808 22:25:47.061606 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0808 22:25:47.061610 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:25:47.061696 (+ 86us) server_negotiation.cc:407] Connection header received
0808 22:25:47.061804 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:25:47.061807 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:25:47.061851 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:25:47.061910 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:25:47.062156 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:47.062663 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:47.063779 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:47.064386 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:47.064490 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:47.064542 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0808 22:25:47.064719 (+ 177us) server_negotiation.cc:299] Negotiation successful
0808 22:25:47.064785 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260808 22:25:47.064914 3858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:47.061415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.193:41243 (local address 127.9.176.194:60083)
0808 22:25:47.061611 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:47.061625 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 22:25:47.061710 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:25:47.061930 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:25:47.061932 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:25:47.061949 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:25:47.062054 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:47.062059 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:47.062763 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:47.062766 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:47.063652 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:47.063664 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:47.064490 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:47.064494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:47.064598 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:47.064621 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 22:25:47.064692 (+ 71us) client_negotiation.cc:240] Negotiation successful
0808 22:25:47.064760 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":2}
I20260808 22:25:47.065358 4079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:47.061560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.194:45197 (local address 127.9.176.195:37195)
0808 22:25:47.061677 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0808 22:25:47.061681 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:25:47.061871 (+ 190us) server_negotiation.cc:407] Connection header received
0808 22:25:47.061981 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:25:47.061983 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:25:47.062018 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:25:47.062075 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:25:47.062342 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:47.062812 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:47.064242 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:47.064799 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:47.064925 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:47.064973 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0808 22:25:47.065193 (+ 220us) server_negotiation.cc:299] Negotiation successful
0808 22:25:47.065262 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 22:25:47.065367 4095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:47.061448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.195:37195 (local address 127.9.176.194:45197)
0808 22:25:47.061788 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:47.061799 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 22:25:47.061891 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:25:47.062091 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:25:47.062094 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:25:47.062109 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:25:47.062244 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:47.062250 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:47.062909 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:47.062913 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:47.064072 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:47.064083 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:47.064952 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:47.064956 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:47.065047 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:47.065070 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 22:25:47.065171 (+ 101us) client_negotiation.cc:240] Negotiation successful
0808 22:25:47.065235 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"mutex_wait_us":30,"thread_start_us":87,"threads_started":1}
I20260808 22:25:47.065521 3586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef2175d7105849e49961acc37682657f" candidate_uuid: "b797dba267504682bfff47ea3a4d7a53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb0243a710734e6f9989e10dac4f70aa" is_pre_election: true
I20260808 22:25:47.065807 3586 raft_consensus.cc:2454] T ef2175d7105849e49961acc37682657f P eb0243a710734e6f9989e10dac4f70aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b797dba267504682bfff47ea3a4d7a53 in term 0.
I20260808 22:25:47.065894 3982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef2175d7105849e49961acc37682657f" candidate_uuid: "b797dba267504682bfff47ea3a4d7a53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4317cf1cf08843ce886bdc9e7500c64e" is_pre_election: true
I20260808 22:25:47.066134 3982 raft_consensus.cc:2454] T ef2175d7105849e49961acc37682657f P 4317cf1cf08843ce886bdc9e7500c64e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b797dba267504682bfff47ea3a4d7a53 in term 0.
I20260808 22:25:47.066262 3696 leader_election.cc:302] T ef2175d7105849e49961acc37682657f P b797dba267504682bfff47ea3a4d7a53 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b797dba267504682bfff47ea3a4d7a53, eb0243a710734e6f9989e10dac4f70aa; no voters:
I20260808 22:25:47.066520 4092 raft_consensus.cc:2781] T ef2175d7105849e49961acc37682657f P b797dba267504682bfff47ea3a4d7a53 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 22:25:47.066601 4092 raft_consensus.cc:482] T ef2175d7105849e49961acc37682657f P b797dba267504682bfff47ea3a4d7a53 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 22:25:47.066632 4092 raft_consensus.cc:3037] T ef2175d7105849e49961acc37682657f P b797dba267504682bfff47ea3a4d7a53 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:25:47.067595 4092 raft_consensus.cc:504] T ef2175d7105849e49961acc37682657f P b797dba267504682bfff47ea3a4d7a53 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0243a710734e6f9989e10dac4f70aa" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 41243 } } peers { permanent_uuid: "4317cf1cf08843ce886bdc9e7500c64e" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37195 } } peers { permanent_uuid: "b797dba267504682bfff47ea3a4d7a53" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 44283 } }
I20260808 22:25:47.067891 4092 leader_election.cc:288] T ef2175d7105849e49961acc37682657f P b797dba267504682bfff47ea3a4d7a53 [CANDIDATE]: Term 1 election: Requested vote from peers eb0243a710734e6f9989e10dac4f70aa (127.9.176.193:41243), 4317cf1cf08843ce886bdc9e7500c64e (127.9.176.195:37195)
I20260808 22:25:47.068192 3982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef2175d7105849e49961acc37682657f" candidate_uuid: "b797dba267504682bfff47ea3a4d7a53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4317cf1cf08843ce886bdc9e7500c64e"
I20260808 22:25:47.068208 3586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef2175d7105849e49961acc37682657f" candidate_uuid: "b797dba267504682bfff47ea3a4d7a53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb0243a710734e6f9989e10dac4f70aa"
I20260808 22:25:47.068288 3982 raft_consensus.cc:3037] T ef2175d7105849e49961acc37682657f P 4317cf1cf08843ce886bdc9e7500c64e [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:25:47.068307 3586 raft_consensus.cc:3037] T ef2175d7105849e49961acc37682657f P eb0243a710734e6f9989e10dac4f70aa [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:25:47.069304 3586 raft_consensus.cc:2454] T ef2175d7105849e49961acc37682657f P eb0243a710734e6f9989e10dac4f70aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b797dba267504682bfff47ea3a4d7a53 in term 1.
I20260808 22:25:47.069336 3982 raft_consensus.cc:2454] T ef2175d7105849e49961acc37682657f P 4317cf1cf08843ce886bdc9e7500c64e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b797dba267504682bfff47ea3a4d7a53 in term 1.
I20260808 22:25:47.069664 3695 leader_election.cc:302] T ef2175d7105849e49961acc37682657f P b797dba267504682bfff47ea3a4d7a53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4317cf1cf08843ce886bdc9e7500c64e, b797dba267504682bfff47ea3a4d7a53; no voters:
I20260808 22:25:47.069818 4092 raft_consensus.cc:2781] T ef2175d7105849e49961acc37682657f P b797dba267504682bfff47ea3a4d7a53 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:25:47.069996 4092 raft_consensus.cc:686] T ef2175d7105849e49961acc37682657f P b797dba267504682bfff47ea3a4d7a53 [term 1 LEADER]: Becoming Leader. State: Replica: b797dba267504682bfff47ea3a4d7a53, State: Running, Role: LEADER
I20260808 22:25:47.070297 4092 consensus_queue.cc:237] T ef2175d7105849e49961acc37682657f P b797dba267504682bfff47ea3a4d7a53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0243a710734e6f9989e10dac4f70aa" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 41243 } } peers { permanent_uuid: "4317cf1cf08843ce886bdc9e7500c64e" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37195 } } peers { permanent_uuid: "b797dba267504682bfff47ea3a4d7a53" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 44283 } }
I20260808 22:25:47.073069 3374 catalog_manager.cc:5495] T ef2175d7105849e49961acc37682657f P b797dba267504682bfff47ea3a4d7a53 reported cstate change: term changed from 0 to 1, leader changed from <none> to b797dba267504682bfff47ea3a4d7a53 (127.9.176.194). New cstate: current_term: 1 leader_uuid: "b797dba267504682bfff47ea3a4d7a53" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0243a710734e6f9989e10dac4f70aa" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 41243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4317cf1cf08843ce886bdc9e7500c64e" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b797dba267504682bfff47ea3a4d7a53" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 44283 } health_report { overall_health: HEALTHY } } }
I20260808 22:25:47.083294 4081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:47.080310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50938 (local address 127.9.176.194:44283)
0808 22:25:47.080411 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0808 22:25:47.080414 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:25:47.080426 (+ 12us) server_negotiation.cc:407] Connection header received
0808 22:25:47.080581 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:25:47.080583 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:25:47.080632 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:25:47.080710 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:25:47.080904 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:47.081389 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:47.082213 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:47.082410 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:47.082474 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:47.082671 (+ 197us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:25:47.083031 (+ 360us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:25:47.083122 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0808 22:25:47.083163 (+ 41us) server_negotiation.cc:299] Negotiation successful
0808 22:25:47.083207 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260808 22:25:47.088394 3586 raft_consensus.cc:1261] T ef2175d7105849e49961acc37682657f P eb0243a710734e6f9989e10dac4f70aa [term 1 FOLLOWER]: Refusing update from remote peer b797dba267504682bfff47ea3a4d7a53: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:25:47.088393 3982 raft_consensus.cc:1261] T ef2175d7105849e49961acc37682657f P 4317cf1cf08843ce886bdc9e7500c64e [term 1 FOLLOWER]: Refusing update from remote peer b797dba267504682bfff47ea3a4d7a53: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:25:47.089005 4096 consensus_queue.cc:1035] T ef2175d7105849e49961acc37682657f P b797dba267504682bfff47ea3a4d7a53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4317cf1cf08843ce886bdc9e7500c64e" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:25:47.089192 4092 consensus_queue.cc:1035] T ef2175d7105849e49961acc37682657f P b797dba267504682bfff47ea3a4d7a53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb0243a710734e6f9989e10dac4f70aa" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 41243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:25:47.096053 4102 mvcc.cc:205] Tried to move back new op lower bound from 7316389671268069376 to 7316389671202373632. Current Snapshot: MvccSnapshot[applied={T|T < 7316389671268069376}]
I20260808 22:25:47.097386 4105 mvcc.cc:205] Tried to move back new op lower bound from 7316389671268069376 to 7316389671202373632. Current Snapshot: MvccSnapshot[applied={T|T < 7316389671268069376}]
I20260808 22:25:47.104220 3373 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38920:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:25:47.104357 3373 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38920:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:25:47.106424 3373 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 21e1c0d6a2b141b89bb4ffbdd8bc410d: Sending DeleteTablet for 3 replicas of tablet ef2175d7105849e49961acc37682657f
I20260808 22:25:47.107112 3954 tablet_service.cc:1514] Processing DeleteTablet for tablet ef2175d7105849e49961acc37682657f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:25:47 UTC) from {username='kudu', principal='kudu/127.9.176.254@KRBTEST.COM'} at 127.0.0.1:34056
I20260808 22:25:47.107143 3561 tablet_service.cc:1514] Processing DeleteTablet for tablet ef2175d7105849e49961acc37682657f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:25:47 UTC) from {username='kudu', principal='kudu/127.9.176.254@KRBTEST.COM'} at 127.0.0.1:50392
I20260808 22:25:47.107262 3762 tablet_service.cc:1514] Processing DeleteTablet for tablet ef2175d7105849e49961acc37682657f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:25:47 UTC) from {username='kudu', principal='kudu/127.9.176.254@KRBTEST.COM'} at 127.0.0.1:50928
I20260808 22:25:47.107671 4117 tablet_replica.cc:330] T ef2175d7105849e49961acc37682657f P b797dba267504682bfff47ea3a4d7a53: stopping tablet replica
I20260808 22:25:47.107694 4115 tablet_replica.cc:330] T ef2175d7105849e49961acc37682657f P 4317cf1cf08843ce886bdc9e7500c64e: stopping tablet replica
I20260808 22:25:47.108017 4117 raft_consensus.cc:2229] T ef2175d7105849e49961acc37682657f P b797dba267504682bfff47ea3a4d7a53 [term 1 LEADER]: Raft consensus shutting down.
I20260808 22:25:47.108070 4115 raft_consensus.cc:2229] T ef2175d7105849e49961acc37682657f P 4317cf1cf08843ce886bdc9e7500c64e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:25:47.108243 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 3464
I20260808 22:25:47.122145 4115 raft_consensus.cc:2258] T ef2175d7105849e49961acc37682657f P 4317cf1cf08843ce886bdc9e7500c64e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:25:47.124200 4115 ts_tablet_manager.cc:1905] T ef2175d7105849e49961acc37682657f P 4317cf1cf08843ce886bdc9e7500c64e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:25:47.126745 4115 ts_tablet_manager.cc:1918] T ef2175d7105849e49961acc37682657f P 4317cf1cf08843ce886bdc9e7500c64e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 22:25:47.126904 4115 log.cc:1196] T ef2175d7105849e49961acc37682657f P 4317cf1cf08843ce886bdc9e7500c64e: Deleting WAL directory at /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/minicluster-data/ts-2/wal/wals/ef2175d7105849e49961acc37682657f
I20260808 22:25:47.127296 4115 ts_tablet_manager.cc:1939] T ef2175d7105849e49961acc37682657f P 4317cf1cf08843ce886bdc9e7500c64e: Deleting consensus metadata
I20260808 22:25:47.128415 3353 catalog_manager.cc:4841] TS 4317cf1cf08843ce886bdc9e7500c64e (127.9.176.195:37195): tablet ef2175d7105849e49961acc37682657f (table test-table [id=1fbfbed10aa3496fa07a4009c1cf5667]) successfully deleted
I20260808 22:25:47.133399 4117 raft_consensus.cc:2258] T ef2175d7105849e49961acc37682657f P b797dba267504682bfff47ea3a4d7a53 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:25:47.135149 4117 ts_tablet_manager.cc:1905] T ef2175d7105849e49961acc37682657f P b797dba267504682bfff47ea3a4d7a53: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:25:47.137274 4117 ts_tablet_manager.cc:1918] T ef2175d7105849e49961acc37682657f P b797dba267504682bfff47ea3a4d7a53: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 22:25:47.137355 4117 log.cc:1196] T ef2175d7105849e49961acc37682657f P b797dba267504682bfff47ea3a4d7a53: Deleting WAL directory at /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786227706282457-9923-0/minicluster-data/ts-1/wal/wals/ef2175d7105849e49961acc37682657f
I20260808 22:25:47.137650 4117 ts_tablet_manager.cc:1939] T ef2175d7105849e49961acc37682657f P b797dba267504682bfff47ea3a4d7a53: Deleting consensus metadata
I20260808 22:25:47.138599 3355 catalog_manager.cc:4841] TS b797dba267504682bfff47ea3a4d7a53 (127.9.176.194:44283): tablet ef2175d7105849e49961acc37682657f (table test-table [id=1fbfbed10aa3496fa07a4009c1cf5667]) successfully deleted
W20260808 22:25:47.144353 3354 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.176.193:41243 (error 108)
I20260808 22:25:47.144583 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 3669
I20260808 22:25:47.148419 4077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:47.148043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.193:41243 (local address 127.0.0.1:50404)
0808 22:25:47.148217 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:47.148315 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.193:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260808 22:25:47.148751 3354 catalog_manager.cc:4584] TS eb0243a710734e6f9989e10dac4f70aa (127.9.176.193:41243): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ef2175d7105849e49961acc37682657f: Network error: Client connection negotiation failed: client connection to 127.9.176.193:41243: connect: Connection refused (error 111)
I20260808 22:25:47.153855 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 3863
I20260808 22:25:47.168821 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 3330
2026-08-08T22:25:47Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2312 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-08T22:25:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:25:47Z Disabled control of system clock
I20260808 22:25:47.201443 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786227706282457-9923-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786227706282457-9923-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786227706282457-9923-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:40201
--webserver_interface=127.9.176.254
--webserver_port=0
--builtin_ntp_servers=127.9.176.212:43799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.176.254:40201
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 22:25:47.311985 4126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:25:47.312220 4126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:25:47.312252 4126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:25:47.316082 4126 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:25:47.316139 4126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:25:47.316152 4126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:25:47.316167 4126 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:25:47.316180 4126 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:25:47.320698 4126 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:43799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786227706282457-9923-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786227706282457-9923-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.176.254:40201
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:40201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.176.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786227706282457-9923-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:25:47.321674 4126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:25:47.322952 4126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:25:47.330945 4133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:25:47.330978 4137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:25:47.331048 4126 server_base.cc:1034] running on GCE node
W20260808 22:25:47.330955 4132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:25:47.331660 4126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:25:47.332675 4126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:25:47.333837 4126 hybrid_clock.cc:648] HybridClock initialized: now 1786227947333821 us; error 26 us; skew 500 ppm
I20260808 22:25:47.335562 4126 webserver.cc:460] Webserver started at http://127.9.176.254:41191/ using document root <none> and password file <none>
I20260808 22:25:47.336107 4126 fs_manager.cc:359] Metadata directory not provided
I20260808 22:25:47.336175 4126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:25:47.336364 4126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:25:47.338466 4126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786227706282457-9923-0/minicluster-data/master-0/data/instance:
uuid: "b9afa44378d84780a27d4c7dc2c74fe9"
format_stamp: "Formatted at 2026-08-08 22:25:47 on dist-test-slave-ncp9"
server_key: "fe6eeb1459a53ed1a9f2c0d41c125dd2"
server_key_iv: "addfa0310e0eba55fbaf71bf16c73ba4"
server_key_version: "encryptionkey@0"
I20260808 22:25:47.338994 4126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786227706282457-9923-0/minicluster-data/master-0/wal/instance:
uuid: "b9afa44378d84780a27d4c7dc2c74fe9"
format_stamp: "Formatted at 2026-08-08 22:25:47 on dist-test-slave-ncp9"
server_key: "fe6eeb1459a53ed1a9f2c0d41c125dd2"
server_key_iv: "addfa0310e0eba55fbaf71bf16c73ba4"
server_key_version: "encryptionkey@0"
I20260808 22:25:47.342537 4126 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 22:25:47.344724 4147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:25:47.345852 4126 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 22:25:47.345985 4126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786227706282457-9923-0/minicluster-data/master-0/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786227706282457-9923-0/minicluster-data/master-0/wal
uuid: "b9afa44378d84780a27d4c7dc2c74fe9"
format_stamp: "Formatted at 2026-08-08 22:25:47 on dist-test-slave-ncp9"
server_key: "fe6eeb1459a53ed1a9f2c0d41c125dd2"
server_key_iv: "addfa0310e0eba55fbaf71bf16c73ba4"
server_key_version: "encryptionkey@0"
I20260808 22:25:47.346076 4126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786227706282457-9923-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786227706282457-9923-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786227706282457-9923-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:25:47.362008 4126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:25:47.362632 4126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:25:47.362789 4126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:25:47.369258 4126 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.254:40201
I20260808 22:25:47.369261 4227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.254:40201 every 8 connection(s)
I20260808 22:25:47.370203 4126 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
I20260808 22:25:47.373189 4228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:25:47.378382 4228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9afa44378d84780a27d4c7dc2c74fe9: Bootstrap starting.
I20260808 22:25:47.378553 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 4126
I20260808 22:25:47.378676 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786227706282457-9923-0/minicluster-data/master-0/wal/instance
I20260808 22:25:47.378923 9923 external_mini_cluster.cc:1428] Setting key d444c13e738f14fb83d8eafe363877f8
I20260808 22:25:47.380880 4228 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9afa44378d84780a27d4c7dc2c74fe9: Neither blocks nor log segments found. Creating new log.
I20260808 22:25:47.381784 4228 log.cc:824] T 00000000000000000000000000000000 P b9afa44378d84780a27d4c7dc2c74fe9: Log is configured to *not* fsync() on all Append() calls
I20260808 22:25:47.383824 4228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9afa44378d84780a27d4c7dc2c74fe9: No bootstrap required, opened a new log
I20260808 22:25:47.387903 4228 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9afa44378d84780a27d4c7dc2c74fe9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9afa44378d84780a27d4c7dc2c74fe9" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 40201 } }
I20260808 22:25:47.388159 4228 raft_consensus.cc:374] T 00000000000000000000000000000000 P b9afa44378d84780a27d4c7dc2c74fe9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:25:47.388226 4228 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9afa44378d84780a27d4c7dc2c74fe9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9afa44378d84780a27d4c7dc2c74fe9, State: Initialized, Role: FOLLOWER
I20260808 22:25:47.388675 4228 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9afa44378d84780a27d4c7dc2c74fe9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9afa44378d84780a27d4c7dc2c74fe9" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 40201 } }
I20260808 22:25:47.388821 4228 raft_consensus.cc:388] T 00000000000000000000000000000000 P b9afa44378d84780a27d4c7dc2c74fe9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:25:47.388886 4228 raft_consensus.cc:482] T 00000000000000000000000000000000 P b9afa44378d84780a27d4c7dc2c74fe9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:25:47.389000 4228 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9afa44378d84780a27d4c7dc2c74fe9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:25:47.389106 4231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:47.380246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59992 (local address 127.9.176.254:40201)
0808 22:25:47.380911 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0808 22:25:47.380933 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0808 22:25:47.380978 (+ 45us) server_negotiation.cc:407] Connection header received
0808 22:25:47.381782 (+ 804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:25:47.381806 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:25:47.382288 (+ 482us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:25:47.382584 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:25:47.382989 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:47.384404 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:47.385415 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:47.386115 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:47.386275 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:47.387147 (+ 872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:25:47.387154 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:25:47.387170 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:25:47.387631 (+ 461us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:25:47.387700 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:25:47.387724 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:25:47.387820 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0808 22:25:47.387907 (+ 87us) server_negotiation.cc:299] Negotiation successful
0808 22:25:47.388170 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":97,"threads_started":1}
I20260808 22:25:47.389847 4228 raft_consensus.cc:504] T 00000000000000000000000000000000 P b9afa44378d84780a27d4c7dc2c74fe9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9afa44378d84780a27d4c7dc2c74fe9" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 40201 } }
I20260808 22:25:47.390157 4228 leader_election.cc:302] T 00000000000000000000000000000000 P b9afa44378d84780a27d4c7dc2c74fe9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b9afa44378d84780a27d4c7dc2c74fe9; no voters:
I20260808 22:25:47.390414 4228 leader_election.cc:288] T 00000000000000000000000000000000 P b9afa44378d84780a27d4c7dc2c74fe9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:25:47.390496 4233 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b9afa44378d84780a27d4c7dc2c74fe9 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:25:47.390713 4233 raft_consensus.cc:686] T 00000000000000000000000000000000 P b9afa44378d84780a27d4c7dc2c74fe9 [term 1 LEADER]: Becoming Leader. State: Replica: b9afa44378d84780a27d4c7dc2c74fe9, State: Running, Role: LEADER
I20260808 22:25:47.391068 4233 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9afa44378d84780a27d4c7dc2c74fe9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9afa44378d84780a27d4c7dc2c74fe9" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 40201 } }
I20260808 22:25:47.391554 4228 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9afa44378d84780a27d4c7dc2c74fe9 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:25:47.392184 4233 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9afa44378d84780a27d4c7dc2c74fe9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9afa44378d84780a27d4c7dc2c74fe9. Latest consensus state: current_term: 1 leader_uuid: "b9afa44378d84780a27d4c7dc2c74fe9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9afa44378d84780a27d4c7dc2c74fe9" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 40201 } } }
I20260808 22:25:47.392378 4233 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9afa44378d84780a27d4c7dc2c74fe9 [sys.catalog]: This master's current role is: LEADER
I20260808 22:25:47.392532 4235 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9afa44378d84780a27d4c7dc2c74fe9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b9afa44378d84780a27d4c7dc2c74fe9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9afa44378d84780a27d4c7dc2c74fe9" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 40201 } } }
I20260808 22:25:47.392692 4235 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9afa44378d84780a27d4c7dc2c74fe9 [sys.catalog]: This master's current role is: LEADER
I20260808 22:25:47.392781 4247 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:25:47.395391 4247 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:25:47.400657 4247 catalog_manager.cc:1347] Generated new cluster ID: 169602ac1a574453b27050eeb4f0b43a
I20260808 22:25:47.400718 4247 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:25:47.409528 4247 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:25:47.410346 4247 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:25:47.413475 4247 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b9afa44378d84780a27d4c7dc2c74fe9: Generated new TSK 0
I20260808 22:25:47.414126 4247 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 22:25:47.416383 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786227706282457-9923-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786227706282457-9923-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786227706282457-9923-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.193:0
--local_ip_for_outbound_sockets=127.9.176.193
--webserver_interface=127.9.176.193
--webserver_port=0
--tserver_master_addrs=127.9.176.254:40201
--builtin_ntp_servers=127.9.176.212:43799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 22:25:47.529893 4259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:25:47.530128 4259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:25:47.530171 4259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:25:47.534132 4259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:25:47.534186 4259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:25:47.534262 4259 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.193
I20260808 22:25:47.538856 4259 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:43799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786227706282457-9923-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786227706282457-9923-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.176.193
--webserver_port=0
--tserver_master_addrs=127.9.176.254:40201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4259
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.193
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786227706282457-9923-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:25:47.539806 4259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:25:47.541179 4259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:25:47.550401 4266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:25:47.550414 4268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:25:47.550626 4259 server_base.cc:1034] running on GCE node
W20260808 22:25:47.550459 4265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:25:47.551095 4259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:25:47.551728 4259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:25:47.552935 4259 hybrid_clock.cc:648] HybridClock initialized: now 1786227947552915 us; error 30 us; skew 500 ppm
I20260808 22:25:47.554692 4259 webserver.cc:460] Webserver started at http://127.9.176.193:46541/ using document root <none> and password file <none>
I20260808 22:25:47.555226 4259 fs_manager.cc:359] Metadata directory not provided
I20260808 22:25:47.555296 4259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:25:47.555511 4259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:25:47.557591 4259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786227706282457-9923-0/minicluster-data/ts-0/data/instance:
uuid: "7bea794f605342e990bf286502cc0f8f"
format_stamp: "Formatted at 2026-08-08 22:25:47 on dist-test-slave-ncp9"
server_key: "f12e0b4326f33b2b690a141d07e829db"
server_key_iv: "403f2dcd6a6d11b24934b21cbeb46004"
server_key_version: "encryptionkey@0"
I20260808 22:25:47.558151 4259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786227706282457-9923-0/minicluster-data/ts-0/wal/instance:
uuid: "7bea794f605342e990bf286502cc0f8f"
format_stamp: "Formatted at 2026-08-08 22:25:47 on dist-test-slave-ncp9"
server_key: "f12e0b4326f33b2b690a141d07e829db"
server_key_iv: "403f2dcd6a6d11b24934b21cbeb46004"
server_key_version: "encryptionkey@0"
I20260808 22:25:47.561781 4259 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 22:25:47.564119 4280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:25:47.565299 4259 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 22:25:47.565410 4259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786227706282457-9923-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786227706282457-9923-0/minicluster-data/ts-0/wal
uuid: "7bea794f605342e990bf286502cc0f8f"
format_stamp: "Formatted at 2026-08-08 22:25:47 on dist-test-slave-ncp9"
server_key: "f12e0b4326f33b2b690a141d07e829db"
server_key_iv: "403f2dcd6a6d11b24934b21cbeb46004"
server_key_version: "encryptionkey@0"
I20260808 22:25:47.565513 4259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786227706282457-9923-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786227706282457-9923-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786227706282457-9923-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:25:47.579640 4259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:25:47.580237 4259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:25:47.580408 4259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:25:47.581019 4259 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:25:47.581920 4259 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:25:47.581976 4259 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:25:47.582032 4259 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:25:47.582060 4259 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:25:47.590968 4259 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.193:46071
I20260808 22:25:47.590996 4461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.193:46071 every 8 connection(s)
I20260808 22:25:47.591830 4259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
I20260808 22:25:47.592729 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 4259
I20260808 22:25:47.592837 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786227706282457-9923-0/minicluster-data/ts-0/wal/instance
I20260808 22:25:47.593169 9923 external_mini_cluster.cc:1428] Setting key db0421690cd9110143203e372dc203f1
I20260808 22:25:47.595278 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786227706282457-9923-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786227706282457-9923-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786227706282457-9923-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.194:0
--local_ip_for_outbound_sockets=127.9.176.194
--webserver_interface=127.9.176.194
--webserver_port=0
--tserver_master_addrs=127.9.176.254:40201
--builtin_ntp_servers=127.9.176.212:43799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 22:25:47.601938 4231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:47.593685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.193:54025 (local address 127.9.176.254:40201)
0808 22:25:47.593929 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0808 22:25:47.593933 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:25:47.595082 (+ 1149us) server_negotiation.cc:407] Connection header received
0808 22:25:47.596881 (+ 1799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:25:47.596886 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:25:47.597005 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:25:47.597101 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:25:47.598031 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:47.598831 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:47.599946 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:47.600290 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:47.600450 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:47.601441 (+ 991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:25:47.601446 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:25:47.601449 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:25:47.601473 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:25:47.601499 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:25:47.601511 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:25:47.601599 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0808 22:25:47.601708 (+ 109us) server_negotiation.cc:299] Negotiation successful
0808 22:25:47.601773 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180}
I20260808 22:25:47.602648 4464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:47.594184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:40201 (local address 127.9.176.193:54025)
0808 22:25:47.594922 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:47.594974 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 22:25:47.596557 (+ 1583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:25:47.597298 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:25:47.597306 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:25:47.597411 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:25:47.597889 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:47.597903 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:47.598949 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:47.598952 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:47.599819 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:47.599828 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:47.600448 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:47.600450 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:47.600598 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:47.601187 (+ 589us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 22:25:47.601189 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:25:47.601227 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 22:25:47.601288 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:25:47.601594 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:25:47.601598 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:25:47.601602 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 22:25:47.601686 (+ 84us) client_negotiation.cc:240] Negotiation successful
0808 22:25:47.601886 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":150,"threads_started":1}
I20260808 22:25:47.603780 4462 heartbeater.cc:347] Connected to a master server at 127.9.176.254:40201
I20260808 22:25:47.604033 4462 heartbeater.cc:464] Registering TS with master...
I20260808 22:25:47.604653 4462 heartbeater.cc:511] Master 127.9.176.254:40201 requested a full tablet report, sending...
I20260808 22:25:47.606194 4173 ts_manager.cc:194] Registered new tserver with Master: 7bea794f605342e990bf286502cc0f8f (127.9.176.193:46071)
I20260808 22:25:47.607281 4173 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.176.193:54025
W20260808 22:25:47.710843 4465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:25:47.711115 4465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:25:47.711154 4465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:25:47.715152 4465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:25:47.715212 4465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:25:47.715277 4465 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.194
I20260808 22:25:47.719959 4465 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:43799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786227706282457-9923-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786227706282457-9923-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.176.194
--webserver_port=0
--tserver_master_addrs=127.9.176.254:40201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4465
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.194
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786227706282457-9923-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:25:47.721058 4465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:25:47.722466 4465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:25:47.732502 4471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:25:47.732470 4473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:25:47.732496 4470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:25:47.733058 4465 server_base.cc:1034] running on GCE node
I20260808 22:25:47.733451 4465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:25:47.734081 4465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:25:47.735273 4465 hybrid_clock.cc:648] HybridClock initialized: now 1786227947735251 us; error 30 us; skew 500 ppm
I20260808 22:25:47.737118 4465 webserver.cc:460] Webserver started at http://127.9.176.194:44785/ using document root <none> and password file <none>
I20260808 22:25:47.737682 4465 fs_manager.cc:359] Metadata directory not provided
I20260808 22:25:47.737751 4465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:25:47.737941 4465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:25:47.739930 4465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786227706282457-9923-0/minicluster-data/ts-1/data/instance:
uuid: "d8727ed594a04ad8b5157c5e7d42d3d2"
format_stamp: "Formatted at 2026-08-08 22:25:47 on dist-test-slave-ncp9"
server_key: "c2f2f3013b61ba1072ba8b5e8b629e41"
server_key_iv: "2cffea8aa673eb0d55b245c0f7d76b45"
server_key_version: "encryptionkey@0"
I20260808 22:25:47.740489 4465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786227706282457-9923-0/minicluster-data/ts-1/wal/instance:
uuid: "d8727ed594a04ad8b5157c5e7d42d3d2"
format_stamp: "Formatted at 2026-08-08 22:25:47 on dist-test-slave-ncp9"
server_key: "c2f2f3013b61ba1072ba8b5e8b629e41"
server_key_iv: "2cffea8aa673eb0d55b245c0f7d76b45"
server_key_version: "encryptionkey@0"
I20260808 22:25:47.744082 4465 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260808 22:25:47.746439 4485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:25:47.747640 4465 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 22:25:47.747740 4465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786227706282457-9923-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786227706282457-9923-0/minicluster-data/ts-1/wal
uuid: "d8727ed594a04ad8b5157c5e7d42d3d2"
format_stamp: "Formatted at 2026-08-08 22:25:47 on dist-test-slave-ncp9"
server_key: "c2f2f3013b61ba1072ba8b5e8b629e41"
server_key_iv: "2cffea8aa673eb0d55b245c0f7d76b45"
server_key_version: "encryptionkey@0"
I20260808 22:25:47.747849 4465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786227706282457-9923-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786227706282457-9923-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786227706282457-9923-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:25:47.767192 4465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:25:47.767860 4465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:25:47.768035 4465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:25:47.768637 4465 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:25:47.769640 4465 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:25:47.769687 4465 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:25:47.769726 4465 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:25:47.769747 4465 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:25:47.778112 4465 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.194:43029
I20260808 22:25:47.778146 4648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.194:43029 every 8 connection(s)
I20260808 22:25:47.778926 4465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
I20260808 22:25:47.783309 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 4465
I20260808 22:25:47.783423 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786227706282457-9923-0/minicluster-data/ts-1/wal/instance
I20260808 22:25:47.783669 9923 external_mini_cluster.cc:1428] Setting key e8d8d92b114b903a5890a174a148b46b
I20260808 22:25:47.785928 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786227706282457-9923-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786227706282457-9923-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786227706282457-9923-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.195:0
--local_ip_for_outbound_sockets=127.9.176.195
--webserver_interface=127.9.176.195
--webserver_port=0
--tserver_master_addrs=127.9.176.254:40201
--builtin_ntp_servers=127.9.176.212:43799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 22:25:47.788606 4231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:47.780634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.194:48665 (local address 127.9.176.254:40201)
0808 22:25:47.780752 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0808 22:25:47.780756 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:25:47.781787 (+ 1031us) server_negotiation.cc:407] Connection header received
0808 22:25:47.783221 (+ 1434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:25:47.783225 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:25:47.783281 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:25:47.783358 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:25:47.784284 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:47.785008 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:47.786660 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:47.786880 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:47.786959 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:47.788101 (+ 1142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:25:47.788105 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:25:47.788108 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:25:47.788123 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:25:47.788147 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:25:47.788156 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:25:47.788223 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0808 22:25:47.788374 (+ 151us) server_negotiation.cc:299] Negotiation successful
0808 22:25:47.788451 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260808 22:25:47.789451 4651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:47.781007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:40201 (local address 127.9.176.194:48665)
0808 22:25:47.781636 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:47.781676 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 22:25:47.782998 (+ 1322us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:25:47.783542 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:25:47.783550 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:25:47.783632 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:25:47.784128 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:47.784146 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:47.785176 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:47.785184 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:47.786532 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:47.786546 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:47.787031 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:47.787035 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:47.787230 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:47.787833 (+ 603us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 22:25:47.787835 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:25:47.787877 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 22:25:47.787948 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:25:47.788234 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:25:47.788244 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:25:47.788251 (+ 7us) client_negotiation.cc:769] Sending connection context
0808 22:25:47.788364 (+ 113us) client_negotiation.cc:240] Negotiation successful
0808 22:25:47.788634 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":110,"threads_started":1}
I20260808 22:25:47.790541 4649 heartbeater.cc:347] Connected to a master server at 127.9.176.254:40201
I20260808 22:25:47.790766 4649 heartbeater.cc:464] Registering TS with master...
I20260808 22:25:47.791272 4649 heartbeater.cc:511] Master 127.9.176.254:40201 requested a full tablet report, sending...
I20260808 22:25:47.792337 4173 ts_manager.cc:194] Registered new tserver with Master: d8727ed594a04ad8b5157c5e7d42d3d2 (127.9.176.194:43029)
I20260808 22:25:47.792997 4173 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.176.194:48665
W20260808 22:25:47.897292 4652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:25:47.897553 4652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:25:47.897608 4652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:25:47.901569 4652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:25:47.901624 4652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:25:47.901696 4652 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.195
I20260808 22:25:47.906389 4652 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:43799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786227706282457-9923-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786227706282457-9923-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.176.195
--webserver_port=0
--tserver_master_addrs=127.9.176.254:40201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4652
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.195
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786227706282457-9923-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:25:47.907442 4652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:25:47.908833 4652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:25:47.918807 4662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:25:47.918849 4652 server_base.cc:1034] running on GCE node
W20260808 22:25:47.918776 4659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:25:47.918792 4660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:25:47.919519 4652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:25:47.920148 4652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:25:47.921314 4652 hybrid_clock.cc:648] HybridClock initialized: now 1786227947921287 us; error 37 us; skew 500 ppm
I20260808 22:25:47.923071 4652 webserver.cc:460] Webserver started at http://127.9.176.195:38513/ using document root <none> and password file <none>
I20260808 22:25:47.923626 4652 fs_manager.cc:359] Metadata directory not provided
I20260808 22:25:47.923692 4652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:25:47.923889 4652 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:25:47.925967 4652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786227706282457-9923-0/minicluster-data/ts-2/data/instance:
uuid: "d1fe7c7fc4c145828049fd5b35af0997"
format_stamp: "Formatted at 2026-08-08 22:25:47 on dist-test-slave-ncp9"
server_key: "5a8f7a4a5e58e74a37af92a658d89db8"
server_key_iv: "3e163590f5a63bfcda337e2a99c3cd52"
server_key_version: "encryptionkey@0"
I20260808 22:25:47.926517 4652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786227706282457-9923-0/minicluster-data/ts-2/wal/instance:
uuid: "d1fe7c7fc4c145828049fd5b35af0997"
format_stamp: "Formatted at 2026-08-08 22:25:47 on dist-test-slave-ncp9"
server_key: "5a8f7a4a5e58e74a37af92a658d89db8"
server_key_iv: "3e163590f5a63bfcda337e2a99c3cd52"
server_key_version: "encryptionkey@0"
I20260808 22:25:47.930015 4652 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 22:25:47.932224 4670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:25:47.933400 4652 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 22:25:47.933516 4652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786227706282457-9923-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786227706282457-9923-0/minicluster-data/ts-2/wal
uuid: "d1fe7c7fc4c145828049fd5b35af0997"
format_stamp: "Formatted at 2026-08-08 22:25:47 on dist-test-slave-ncp9"
server_key: "5a8f7a4a5e58e74a37af92a658d89db8"
server_key_iv: "3e163590f5a63bfcda337e2a99c3cd52"
server_key_version: "encryptionkey@0"
I20260808 22:25:47.933607 4652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786227706282457-9923-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786227706282457-9923-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786227706282457-9923-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:25:47.947527 4652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:25:47.948134 4652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:25:47.948287 4652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:25:47.948824 4652 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:25:47.949755 4652 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:25:47.949802 4652 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:25:47.949839 4652 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:25:47.949867 4652 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:25:47.958446 4652 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.195:45309
I20260808 22:25:47.958510 4826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.195:45309 every 8 connection(s)
I20260808 22:25:47.959295 4652 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
I20260808 22:25:47.963039 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 4652
I20260808 22:25:47.963124 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786227706282457-9923-0/minicluster-data/ts-2/wal/instance
I20260808 22:25:47.963348 9923 external_mini_cluster.cc:1428] Setting key 70a550607472cd601d85b88c72f2b792
I20260808 22:25:47.968035 4231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:47.960966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.195:40155 (local address 127.9.176.254:40201)
0808 22:25:47.961103 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0808 22:25:47.961109 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 22:25:47.962143 (+ 1034us) server_negotiation.cc:407] Connection header received
0808 22:25:47.963570 (+ 1427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:25:47.963574 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:25:47.963630 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:25:47.963704 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:25:47.964663 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:47.965396 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:47.966482 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:47.966690 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:47.966758 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:47.967695 (+ 937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:25:47.967697 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:25:47.967699 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:25:47.967709 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:25:47.967724 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:25:47.967730 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:25:47.967773 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0808 22:25:47.967876 (+ 103us) server_negotiation.cc:299] Negotiation successful
0808 22:25:47.967922 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260808 22:25:47.968879 4829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:47.961375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:40201 (local address 127.9.176.195:40155)
0808 22:25:47.962001 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:47.962039 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0808 22:25:47.963360 (+ 1321us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:25:47.963891 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:25:47.963898 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:25:47.963979 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:25:47.964559 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:47.964576 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:47.965516 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:47.965519 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:47.966391 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:47.966400 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:47.966793 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:47.966796 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:47.966929 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:47.967475 (+ 546us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 22:25:47.967477 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:25:47.967516 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 22:25:47.967578 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:25:47.967782 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:25:47.967787 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:25:47.967791 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 22:25:47.967872 (+ 81us) client_negotiation.cc:240] Negotiation successful
0808 22:25:47.968106 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":111,"threads_started":1}
I20260808 22:25:47.969977 4827 heartbeater.cc:347] Connected to a master server at 127.9.176.254:40201
I20260808 22:25:47.970196 4827 heartbeater.cc:464] Registering TS with master...
I20260808 22:25:47.970661 4827 heartbeater.cc:511] Master 127.9.176.254:40201 requested a full tablet report, sending...
I20260808 22:25:47.971680 4173 ts_manager.cc:194] Registered new tserver with Master: d1fe7c7fc4c145828049fd5b35af0997 (127.9.176.195:45309)
I20260808 22:25:47.972255 4173 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.176.195:40155
I20260808 22:25:47.976959 9923 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 22:25:47.982322 4231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:47.978833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60006 (local address 127.9.176.254:40201)
0808 22:25:47.978977 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0808 22:25:47.978982 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:25:47.979106 (+ 124us) server_negotiation.cc:407] Connection header received
0808 22:25:47.979218 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:25:47.979221 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:25:47.979265 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:25:47.979328 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:25:47.979600 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:47.980101 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:47.980941 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:47.981160 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:47.981246 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:47.981932 (+ 686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:25:47.981936 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:25:47.981939 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:25:47.981959 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:25:47.981977 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:25:47.981985 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:25:47.982040 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0808 22:25:47.982133 (+ 93us) server_negotiation.cc:299] Negotiation successful
0808 22:25:47.982196 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 22:25:47.984772 4173 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60006:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 22:25:47.987627 4173 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 22:25:48.000952 4844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:47.995262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37638 (local address 127.9.176.195:45309)
0808 22:25:47.995846 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0808 22:25:47.995857 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 22:25:47.996219 (+ 362us) server_negotiation.cc:407] Connection header received
0808 22:25:47.996305 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:25:47.996310 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:25:47.996512 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:25:47.996661 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:25:47.997163 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:47.998054 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:47.999761 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:48.000285 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:48.000381 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:48.000585 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0808 22:25:48.000685 (+ 100us) server_negotiation.cc:299] Negotiation successful
0808 22:25:48.000780 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":125,"threads_started":1}
I20260808 22:25:48.001174 4842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:47.995241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.194:43029 (local address 127.0.0.1:56126)
0808 22:25:47.995905 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:47.995955 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0808 22:25:47.996070 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:25:47.996974 (+ 904us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:25:47.996977 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:25:47.996995 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:25:47.997109 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:47.997116 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:47.998316 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:47.998319 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:47.999614 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:47.999626 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:48.000675 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:48.000678 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:48.000767 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:48.000790 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 22:25:48.000839 (+ 49us) client_negotiation.cc:240] Negotiation successful
0808 22:25:48.000934 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"mutex_wait_us":51,"thread_start_us":102,"threads_started":1}
I20260808 22:25:48.000943 4845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:47.995133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.195:45309 (local address 127.0.0.1:37638)
0808 22:25:47.995952 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:47.995966 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 22:25:47.996071 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:25:47.996835 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:25:47.996840 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:25:47.996863 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:25:47.997024 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:47.997031 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:47.998155 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:47.998159 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:47.999644 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:47.999652 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:48.000402 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:48.000405 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:48.000511 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:48.000549 (+ 38us) client_negotiation.cc:769] Sending connection context
0808 22:25:48.000658 (+ 109us) client_negotiation.cc:240] Negotiation successful
0808 22:25:48.000774 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"mutex_wait_us":90,"thread_start_us":66,"threads_started":1}
I20260808 22:25:48.001477 4846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:47.996041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56126 (local address 127.9.176.194:43029)
0808 22:25:47.996501 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0808 22:25:47.996512 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 22:25:47.996537 (+ 25us) server_negotiation.cc:407] Connection header received
0808 22:25:47.996625 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:25:47.996630 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:25:47.996816 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:25:47.997000 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:25:47.997211 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:47.998205 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:47.999748 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:48.000524 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:48.001013 (+ 489us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:48.001188 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0808 22:25:48.001236 (+ 48us) server_negotiation.cc:299] Negotiation successful
0808 22:25:48.001318 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":166,"threads_started":1}
I20260808 22:25:48.001492 4847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:47.996308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56814 (local address 127.9.176.193:46071)
0808 22:25:47.996671 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0808 22:25:47.996684 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 22:25:47.996769 (+ 85us) server_negotiation.cc:407] Connection header received
0808 22:25:47.996880 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:25:47.996885 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:25:47.997086 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:25:47.997220 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:25:47.997595 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:47.998606 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:47.999778 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:48.000524 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:48.000626 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:48.000798 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0808 22:25:48.001238 (+ 440us) server_negotiation.cc:299] Negotiation successful
0808 22:25:48.001323 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":121,"threads_started":1}
I20260808 22:25:48.001751 4843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:47.995458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.193:46071 (local address 127.0.0.1:56814)
0808 22:25:47.996674 (+ 1216us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:47.996687 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 22:25:47.996775 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:25:47.997359 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:25:47.997361 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:25:47.997378 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:25:47.997463 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:47.997468 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:47.998716 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:47.998719 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:47.999651 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:47.999661 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:48.000856 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:48.000859 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:48.000971 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:48.000993 (+ 22us) client_negotiation.cc:769] Sending connection context
0808 22:25:48.001054 (+ 61us) client_negotiation.cc:240] Negotiation successful
0808 22:25:48.001128 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1160,"mutex_wait_us":60,"thread_start_us":95,"threads_started":1}
I20260808 22:25:48.004432 4552 tablet_service.cc:1467] Processing CreateTablet for tablet 5fac7e4ebab641b3ad2428a751310f59 (DEFAULT_TABLE table=test-table [id=205cb96a738640579e54ad4e9e5b12e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:25:48.004437 4361 tablet_service.cc:1467] Processing CreateTablet for tablet 5fac7e4ebab641b3ad2428a751310f59 (DEFAULT_TABLE table=test-table [id=205cb96a738640579e54ad4e9e5b12e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:25:48.005589 4552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fac7e4ebab641b3ad2428a751310f59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:25:48.005597 4361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fac7e4ebab641b3ad2428a751310f59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:25:48.010716 4737 tablet_service.cc:1467] Processing CreateTablet for tablet 5fac7e4ebab641b3ad2428a751310f59 (DEFAULT_TABLE table=test-table [id=205cb96a738640579e54ad4e9e5b12e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:25:48.011782 4737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fac7e4ebab641b3ad2428a751310f59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:25:48.013144 4849 tablet_bootstrap.cc:492] T 5fac7e4ebab641b3ad2428a751310f59 P d8727ed594a04ad8b5157c5e7d42d3d2: Bootstrap starting.
I20260808 22:25:48.013576 4848 tablet_bootstrap.cc:492] T 5fac7e4ebab641b3ad2428a751310f59 P 7bea794f605342e990bf286502cc0f8f: Bootstrap starting.
I20260808 22:25:48.015962 4848 tablet_bootstrap.cc:654] T 5fac7e4ebab641b3ad2428a751310f59 P 7bea794f605342e990bf286502cc0f8f: Neither blocks nor log segments found. Creating new log.
I20260808 22:25:48.016916 4848 log.cc:824] T 5fac7e4ebab641b3ad2428a751310f59 P 7bea794f605342e990bf286502cc0f8f: Log is configured to *not* fsync() on all Append() calls
I20260808 22:25:48.018707 4849 tablet_bootstrap.cc:654] T 5fac7e4ebab641b3ad2428a751310f59 P d8727ed594a04ad8b5157c5e7d42d3d2: Neither blocks nor log segments found. Creating new log.
I20260808 22:25:48.018915 4852 tablet_bootstrap.cc:492] T 5fac7e4ebab641b3ad2428a751310f59 P d1fe7c7fc4c145828049fd5b35af0997: Bootstrap starting.
I20260808 22:25:48.019029 4848 tablet_bootstrap.cc:492] T 5fac7e4ebab641b3ad2428a751310f59 P 7bea794f605342e990bf286502cc0f8f: No bootstrap required, opened a new log
I20260808 22:25:48.019508 4848 ts_tablet_manager.cc:1397] T 5fac7e4ebab641b3ad2428a751310f59 P 7bea794f605342e990bf286502cc0f8f: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260808 22:25:48.019575 4849 log.cc:824] T 5fac7e4ebab641b3ad2428a751310f59 P d8727ed594a04ad8b5157c5e7d42d3d2: Log is configured to *not* fsync() on all Append() calls
I20260808 22:25:48.021009 4852 tablet_bootstrap.cc:654] T 5fac7e4ebab641b3ad2428a751310f59 P d1fe7c7fc4c145828049fd5b35af0997: Neither blocks nor log segments found. Creating new log.
I20260808 22:25:48.021647 4849 tablet_bootstrap.cc:492] T 5fac7e4ebab641b3ad2428a751310f59 P d8727ed594a04ad8b5157c5e7d42d3d2: No bootstrap required, opened a new log
I20260808 22:25:48.021783 4852 log.cc:824] T 5fac7e4ebab641b3ad2428a751310f59 P d1fe7c7fc4c145828049fd5b35af0997: Log is configured to *not* fsync() on all Append() calls
I20260808 22:25:48.021835 4849 ts_tablet_manager.cc:1397] T 5fac7e4ebab641b3ad2428a751310f59 P d8727ed594a04ad8b5157c5e7d42d3d2: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260808 22:25:48.023165 4848 raft_consensus.cc:348] T 5fac7e4ebab641b3ad2428a751310f59 P 7bea794f605342e990bf286502cc0f8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1fe7c7fc4c145828049fd5b35af0997" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45309 } } peers { permanent_uuid: "d8727ed594a04ad8b5157c5e7d42d3d2" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43029 } } peers { permanent_uuid: "7bea794f605342e990bf286502cc0f8f" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 46071 } }
I20260808 22:25:48.023478 4848 raft_consensus.cc:374] T 5fac7e4ebab641b3ad2428a751310f59 P 7bea794f605342e990bf286502cc0f8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:25:48.023542 4848 raft_consensus.cc:729] T 5fac7e4ebab641b3ad2428a751310f59 P 7bea794f605342e990bf286502cc0f8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bea794f605342e990bf286502cc0f8f, State: Initialized, Role: FOLLOWER
I20260808 22:25:48.023976 4848 consensus_queue.cc:260] T 5fac7e4ebab641b3ad2428a751310f59 P 7bea794f605342e990bf286502cc0f8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1fe7c7fc4c145828049fd5b35af0997" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45309 } } peers { permanent_uuid: "d8727ed594a04ad8b5157c5e7d42d3d2" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43029 } } peers { permanent_uuid: "7bea794f605342e990bf286502cc0f8f" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 46071 } }
I20260808 22:25:48.024053 4852 tablet_bootstrap.cc:492] T 5fac7e4ebab641b3ad2428a751310f59 P d1fe7c7fc4c145828049fd5b35af0997: No bootstrap required, opened a new log
I20260808 22:25:48.024230 4852 ts_tablet_manager.cc:1397] T 5fac7e4ebab641b3ad2428a751310f59 P d1fe7c7fc4c145828049fd5b35af0997: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260808 22:25:48.024789 4848 ts_tablet_manager.cc:1428] T 5fac7e4ebab641b3ad2428a751310f59 P 7bea794f605342e990bf286502cc0f8f: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260808 22:25:48.025156 4462 heartbeater.cc:503] Master 127.9.176.254:40201 was elected leader, sending a full tablet report...
I20260808 22:25:48.025851 4849 raft_consensus.cc:348] T 5fac7e4ebab641b3ad2428a751310f59 P d8727ed594a04ad8b5157c5e7d42d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1fe7c7fc4c145828049fd5b35af0997" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45309 } } peers { permanent_uuid: "d8727ed594a04ad8b5157c5e7d42d3d2" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43029 } } peers { permanent_uuid: "7bea794f605342e990bf286502cc0f8f" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 46071 } }
I20260808 22:25:48.026175 4849 raft_consensus.cc:374] T 5fac7e4ebab641b3ad2428a751310f59 P d8727ed594a04ad8b5157c5e7d42d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:25:48.026242 4849 raft_consensus.cc:729] T 5fac7e4ebab641b3ad2428a751310f59 P d8727ed594a04ad8b5157c5e7d42d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8727ed594a04ad8b5157c5e7d42d3d2, State: Initialized, Role: FOLLOWER
I20260808 22:25:48.026686 4849 consensus_queue.cc:260] T 5fac7e4ebab641b3ad2428a751310f59 P d8727ed594a04ad8b5157c5e7d42d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1fe7c7fc4c145828049fd5b35af0997" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45309 } } peers { permanent_uuid: "d8727ed594a04ad8b5157c5e7d42d3d2" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43029 } } peers { permanent_uuid: "7bea794f605342e990bf286502cc0f8f" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 46071 } }
I20260808 22:25:48.027336 4649 heartbeater.cc:503] Master 127.9.176.254:40201 was elected leader, sending a full tablet report...
I20260808 22:25:48.027619 4849 ts_tablet_manager.cc:1428] T 5fac7e4ebab641b3ad2428a751310f59 P d8727ed594a04ad8b5157c5e7d42d3d2: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260808 22:25:48.028229 4852 raft_consensus.cc:348] T 5fac7e4ebab641b3ad2428a751310f59 P d1fe7c7fc4c145828049fd5b35af0997 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1fe7c7fc4c145828049fd5b35af0997" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45309 } } peers { permanent_uuid: "d8727ed594a04ad8b5157c5e7d42d3d2" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43029 } } peers { permanent_uuid: "7bea794f605342e990bf286502cc0f8f" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 46071 } }
I20260808 22:25:48.028492 4852 raft_consensus.cc:374] T 5fac7e4ebab641b3ad2428a751310f59 P d1fe7c7fc4c145828049fd5b35af0997 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:25:48.028559 4852 raft_consensus.cc:729] T 5fac7e4ebab641b3ad2428a751310f59 P d1fe7c7fc4c145828049fd5b35af0997 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1fe7c7fc4c145828049fd5b35af0997, State: Initialized, Role: FOLLOWER
I20260808 22:25:48.029062 4852 consensus_queue.cc:260] T 5fac7e4ebab641b3ad2428a751310f59 P d1fe7c7fc4c145828049fd5b35af0997 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1fe7c7fc4c145828049fd5b35af0997" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45309 } } peers { permanent_uuid: "d8727ed594a04ad8b5157c5e7d42d3d2" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43029 } } peers { permanent_uuid: "7bea794f605342e990bf286502cc0f8f" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 46071 } }
I20260808 22:25:48.029772 4827 heartbeater.cc:503] Master 127.9.176.254:40201 was elected leader, sending a full tablet report...
W20260808 22:25:48.029773 4650 tablet.cc:2369] T 5fac7e4ebab641b3ad2428a751310f59 P d8727ed594a04ad8b5157c5e7d42d3d2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 22:25:48.029975 4852 ts_tablet_manager.cc:1428] T 5fac7e4ebab641b3ad2428a751310f59 P d1fe7c7fc4c145828049fd5b35af0997: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260808 22:25:48.090294 4857 raft_consensus.cc:482] T 5fac7e4ebab641b3ad2428a751310f59 P d8727ed594a04ad8b5157c5e7d42d3d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 22:25:48.090497 4857 raft_consensus.cc:504] T 5fac7e4ebab641b3ad2428a751310f59 P d8727ed594a04ad8b5157c5e7d42d3d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1fe7c7fc4c145828049fd5b35af0997" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45309 } } peers { permanent_uuid: "d8727ed594a04ad8b5157c5e7d42d3d2" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43029 } } peers { permanent_uuid: "7bea794f605342e990bf286502cc0f8f" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 46071 } }
I20260808 22:25:48.091640 4857 leader_election.cc:288] T 5fac7e4ebab641b3ad2428a751310f59 P d8727ed594a04ad8b5157c5e7d42d3d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1fe7c7fc4c145828049fd5b35af0997 (127.9.176.195:45309), 7bea794f605342e990bf286502cc0f8f (127.9.176.193:46071)
W20260808 22:25:48.093112 4463 tablet.cc:2369] T 5fac7e4ebab641b3ad2428a751310f59 P 7bea794f605342e990bf286502cc0f8f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 22:25:48.095180 4651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:48.091833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.193:46071 (local address 127.9.176.194:41191)
0808 22:25:48.091956 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:48.091973 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 22:25:48.092080 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:25:48.092348 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:25:48.092351 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:25:48.092370 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:25:48.092475 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:48.092480 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:48.093349 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:48.093352 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:48.094129 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:48.094138 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:48.094772 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:48.094776 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:48.094888 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:48.094907 (+ 19us) client_negotiation.cc:769] Sending connection context
0808 22:25:48.094970 (+ 63us) client_negotiation.cc:240] Negotiation successful
0808 22:25:48.095038 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":1}
I20260808 22:25:48.095180 4847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:48.091971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.194:41191 (local address 127.9.176.193:46071)
0808 22:25:48.092095 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0808 22:25:48.092100 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:25:48.092144 (+ 44us) server_negotiation.cc:407] Connection header received
0808 22:25:48.092200 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:25:48.092203 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:25:48.092251 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:25:48.092315 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:25:48.092583 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:48.093219 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:48.094257 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:48.094661 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:48.094765 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:48.094818 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0808 22:25:48.094997 (+ 179us) server_negotiation.cc:299] Negotiation successful
0808 22:25:48.095068 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 22:25:48.095692 4860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:48.091833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.195:45309 (local address 127.9.176.194:51899)
0808 22:25:48.092330 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:48.092341 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 22:25:48.092438 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:25:48.092699 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:25:48.092702 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:25:48.092720 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:25:48.092829 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:48.092834 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:48.093784 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:48.093787 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:48.094584 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:48.094592 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:48.095249 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:48.095253 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:48.095352 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:48.095377 (+ 25us) client_negotiation.cc:769] Sending connection context
0808 22:25:48.095481 (+ 104us) client_negotiation.cc:240] Negotiation successful
0808 22:25:48.095549 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"mutex_wait_us":108,"thread_start_us":160,"threads_started":1}
I20260808 22:25:48.095707 4844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:48.092005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.194:51899 (local address 127.9.176.195:45309)
0808 22:25:48.092182 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0808 22:25:48.092187 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:25:48.092425 (+ 238us) server_negotiation.cc:407] Connection header received
0808 22:25:48.092526 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:25:48.092529 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:25:48.092573 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:25:48.092633 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:25:48.092938 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:48.093637 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:48.094708 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:48.095116 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:48.095225 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:48.095270 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0808 22:25:48.095514 (+ 244us) server_negotiation.cc:299] Negotiation successful
0808 22:25:48.095585 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260808 22:25:48.095914 4392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fac7e4ebab641b3ad2428a751310f59" candidate_uuid: "d8727ed594a04ad8b5157c5e7d42d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bea794f605342e990bf286502cc0f8f" is_pre_election: true
I20260808 22:25:48.096226 4392 raft_consensus.cc:2454] T 5fac7e4ebab641b3ad2428a751310f59 P 7bea794f605342e990bf286502cc0f8f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8727ed594a04ad8b5157c5e7d42d3d2 in term 0.
I20260808 22:25:48.096366 4766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fac7e4ebab641b3ad2428a751310f59" candidate_uuid: "d8727ed594a04ad8b5157c5e7d42d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1fe7c7fc4c145828049fd5b35af0997" is_pre_election: true
I20260808 22:25:48.096661 4766 raft_consensus.cc:2454] T 5fac7e4ebab641b3ad2428a751310f59 P d1fe7c7fc4c145828049fd5b35af0997 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8727ed594a04ad8b5157c5e7d42d3d2 in term 0.
I20260808 22:25:48.096727 4489 leader_election.cc:302] T 5fac7e4ebab641b3ad2428a751310f59 P d8727ed594a04ad8b5157c5e7d42d3d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7bea794f605342e990bf286502cc0f8f, d8727ed594a04ad8b5157c5e7d42d3d2; no voters:
I20260808 22:25:48.097041 4857 raft_consensus.cc:2781] T 5fac7e4ebab641b3ad2428a751310f59 P d8727ed594a04ad8b5157c5e7d42d3d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 22:25:48.097132 4857 raft_consensus.cc:482] T 5fac7e4ebab641b3ad2428a751310f59 P d8727ed594a04ad8b5157c5e7d42d3d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 22:25:48.097182 4857 raft_consensus.cc:3037] T 5fac7e4ebab641b3ad2428a751310f59 P d8727ed594a04ad8b5157c5e7d42d3d2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:25:48.098173 4857 raft_consensus.cc:504] T 5fac7e4ebab641b3ad2428a751310f59 P d8727ed594a04ad8b5157c5e7d42d3d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1fe7c7fc4c145828049fd5b35af0997" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45309 } } peers { permanent_uuid: "d8727ed594a04ad8b5157c5e7d42d3d2" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43029 } } peers { permanent_uuid: "7bea794f605342e990bf286502cc0f8f" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 46071 } }
I20260808 22:25:48.098483 4857 leader_election.cc:288] T 5fac7e4ebab641b3ad2428a751310f59 P d8727ed594a04ad8b5157c5e7d42d3d2 [CANDIDATE]: Term 1 election: Requested vote from peers d1fe7c7fc4c145828049fd5b35af0997 (127.9.176.195:45309), 7bea794f605342e990bf286502cc0f8f (127.9.176.193:46071)
I20260808 22:25:48.098827 4392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fac7e4ebab641b3ad2428a751310f59" candidate_uuid: "d8727ed594a04ad8b5157c5e7d42d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bea794f605342e990bf286502cc0f8f"
I20260808 22:25:48.098935 4392 raft_consensus.cc:3037] T 5fac7e4ebab641b3ad2428a751310f59 P 7bea794f605342e990bf286502cc0f8f [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:25:48.099937 4392 raft_consensus.cc:2454] T 5fac7e4ebab641b3ad2428a751310f59 P 7bea794f605342e990bf286502cc0f8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8727ed594a04ad8b5157c5e7d42d3d2 in term 1.
I20260808 22:25:48.100178 4766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fac7e4ebab641b3ad2428a751310f59" candidate_uuid: "d8727ed594a04ad8b5157c5e7d42d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1fe7c7fc4c145828049fd5b35af0997"
I20260808 22:25:48.100246 4489 leader_election.cc:302] T 5fac7e4ebab641b3ad2428a751310f59 P d8727ed594a04ad8b5157c5e7d42d3d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7bea794f605342e990bf286502cc0f8f, d8727ed594a04ad8b5157c5e7d42d3d2; no voters:
I20260808 22:25:48.100304 4766 raft_consensus.cc:3037] T 5fac7e4ebab641b3ad2428a751310f59 P d1fe7c7fc4c145828049fd5b35af0997 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:25:48.100441 4857 raft_consensus.cc:2781] T 5fac7e4ebab641b3ad2428a751310f59 P d8727ed594a04ad8b5157c5e7d42d3d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:25:48.100674 4857 raft_consensus.cc:686] T 5fac7e4ebab641b3ad2428a751310f59 P d8727ed594a04ad8b5157c5e7d42d3d2 [term 1 LEADER]: Becoming Leader. State: Replica: d8727ed594a04ad8b5157c5e7d42d3d2, State: Running, Role: LEADER
I20260808 22:25:48.101184 4857 consensus_queue.cc:237] T 5fac7e4ebab641b3ad2428a751310f59 P d8727ed594a04ad8b5157c5e7d42d3d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1fe7c7fc4c145828049fd5b35af0997" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45309 } } peers { permanent_uuid: "d8727ed594a04ad8b5157c5e7d42d3d2" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43029 } } peers { permanent_uuid: "7bea794f605342e990bf286502cc0f8f" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 46071 } }
I20260808 22:25:48.101452 4766 raft_consensus.cc:2454] T 5fac7e4ebab641b3ad2428a751310f59 P d1fe7c7fc4c145828049fd5b35af0997 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8727ed594a04ad8b5157c5e7d42d3d2 in term 1.
I20260808 22:25:48.104393 4173 catalog_manager.cc:5495] T 5fac7e4ebab641b3ad2428a751310f59 P d8727ed594a04ad8b5157c5e7d42d3d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to d8727ed594a04ad8b5157c5e7d42d3d2 (127.9.176.194). New cstate: current_term: 1 leader_uuid: "d8727ed594a04ad8b5157c5e7d42d3d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1fe7c7fc4c145828049fd5b35af0997" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8727ed594a04ad8b5157c5e7d42d3d2" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 43029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7bea794f605342e990bf286502cc0f8f" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 46071 } health_report { overall_health: UNKNOWN } } }
I20260808 22:25:48.120587 4846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:48.117660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56136 (local address 127.9.176.194:43029)
0808 22:25:48.117763 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0808 22:25:48.117767 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:25:48.117779 (+ 12us) server_negotiation.cc:407] Connection header received
0808 22:25:48.117842 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:25:48.117844 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:25:48.117893 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:25:48.117974 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:25:48.118156 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:48.118636 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:48.119478 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:48.119673 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:48.119733 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:48.119944 (+ 211us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:25:48.120337 (+ 393us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:25:48.120418 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0808 22:25:48.120446 (+ 28us) server_negotiation.cc:299] Negotiation successful
0808 22:25:48.120488 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260808 22:25:48.126111 4766 raft_consensus.cc:1261] T 5fac7e4ebab641b3ad2428a751310f59 P d1fe7c7fc4c145828049fd5b35af0997 [term 1 FOLLOWER]: Refusing update from remote peer d8727ed594a04ad8b5157c5e7d42d3d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:25:48.126111 4392 raft_consensus.cc:1261] T 5fac7e4ebab641b3ad2428a751310f59 P 7bea794f605342e990bf286502cc0f8f [term 1 FOLLOWER]: Refusing update from remote peer d8727ed594a04ad8b5157c5e7d42d3d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:25:48.126782 4861 consensus_queue.cc:1035] T 5fac7e4ebab641b3ad2428a751310f59 P d8727ed594a04ad8b5157c5e7d42d3d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bea794f605342e990bf286502cc0f8f" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 46071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:25:48.126952 4857 consensus_queue.cc:1035] T 5fac7e4ebab641b3ad2428a751310f59 P d8727ed594a04ad8b5157c5e7d42d3d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1fe7c7fc4c145828049fd5b35af0997" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:25:48.136497 4866 mvcc.cc:205] Tried to move back new op lower bound from 7316389675517792256 to 7316389675425169408. Current Snapshot: MvccSnapshot[applied={T|T < 7316389675517792256}]
I20260808 22:25:48.142021 4171 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:25:48.142211 4171 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:25:48.144506 4171 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b9afa44378d84780a27d4c7dc2c74fe9: Sending DeleteTablet for 3 replicas of tablet 5fac7e4ebab641b3ad2428a751310f59
I20260808 22:25:48.145242 4552 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fac7e4ebab641b3ad2428a751310f59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:25:48 UTC) from {username='slave'} at 127.0.0.1:56126
I20260808 22:25:48.145248 4737 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fac7e4ebab641b3ad2428a751310f59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:25:48 UTC) from {username='slave'} at 127.0.0.1:37638
I20260808 22:25:48.145321 4361 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fac7e4ebab641b3ad2428a751310f59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:25:48 UTC) from {username='slave'} at 127.0.0.1:56814
I20260808 22:25:48.145823 4879 tablet_replica.cc:330] T 5fac7e4ebab641b3ad2428a751310f59 P d8727ed594a04ad8b5157c5e7d42d3d2: stopping tablet replica
I20260808 22:25:48.146117 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 4259
I20260808 22:25:48.146129 4879 raft_consensus.cc:2229] T 5fac7e4ebab641b3ad2428a751310f59 P d8727ed594a04ad8b5157c5e7d42d3d2 [term 1 LEADER]: Raft consensus shutting down.
I20260808 22:25:48.146472 4879 raft_consensus.cc:2258] T 5fac7e4ebab641b3ad2428a751310f59 P d8727ed594a04ad8b5157c5e7d42d3d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:25:48.147025 4880 tablet_replica.cc:330] T 5fac7e4ebab641b3ad2428a751310f59 P d1fe7c7fc4c145828049fd5b35af0997: stopping tablet replica
I20260808 22:25:48.147321 4880 raft_consensus.cc:2229] T 5fac7e4ebab641b3ad2428a751310f59 P d1fe7c7fc4c145828049fd5b35af0997 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:25:48.147617 4880 raft_consensus.cc:2258] T 5fac7e4ebab641b3ad2428a751310f59 P d1fe7c7fc4c145828049fd5b35af0997 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:25:48.157689 4879 ts_tablet_manager.cc:1905] T 5fac7e4ebab641b3ad2428a751310f59 P d8727ed594a04ad8b5157c5e7d42d3d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:25:48.160569 4879 ts_tablet_manager.cc:1918] T 5fac7e4ebab641b3ad2428a751310f59 P d8727ed594a04ad8b5157c5e7d42d3d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 22:25:48.160681 4879 log.cc:1196] T 5fac7e4ebab641b3ad2428a751310f59 P d8727ed594a04ad8b5157c5e7d42d3d2: Deleting WAL directory at /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786227706282457-9923-0/minicluster-data/ts-1/wal/wals/5fac7e4ebab641b3ad2428a751310f59
I20260808 22:25:48.161154 4879 ts_tablet_manager.cc:1939] T 5fac7e4ebab641b3ad2428a751310f59 P d8727ed594a04ad8b5157c5e7d42d3d2: Deleting consensus metadata
I20260808 22:25:48.161809 4880 ts_tablet_manager.cc:1905] T 5fac7e4ebab641b3ad2428a751310f59 P d1fe7c7fc4c145828049fd5b35af0997: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260808 22:25:48.162539 4153 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.176.193:46071 (error 108)
I20260808 22:25:48.162627 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 4465
I20260808 22:25:48.170806 4843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:48.170467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.193:46071 (local address 127.0.0.1:56828)
0808 22:25:48.170657 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:48.170724 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.193:46071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
W20260808 22:25:48.171015 4153 catalog_manager.cc:4584] TS 7bea794f605342e990bf286502cc0f8f (127.9.176.193:46071): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5fac7e4ebab641b3ad2428a751310f59: Network error: Client connection negotiation failed: client connection to 127.9.176.193:46071: connect: Connection refused (error 111)
I20260808 22:25:48.173985 4880 ts_tablet_manager.cc:1918] T 5fac7e4ebab641b3ad2428a751310f59 P d1fe7c7fc4c145828049fd5b35af0997: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 22:25:48.174094 4880 log.cc:1196] T 5fac7e4ebab641b3ad2428a751310f59 P d1fe7c7fc4c145828049fd5b35af0997: Deleting WAL directory at /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786227706282457-9923-0/minicluster-data/ts-2/wal/wals/5fac7e4ebab641b3ad2428a751310f59
I20260808 22:25:48.174374 4880 ts_tablet_manager.cc:1939] T 5fac7e4ebab641b3ad2428a751310f59 P d1fe7c7fc4c145828049fd5b35af0997: Deleting consensus metadata
I20260808 22:25:48.175539 4150 catalog_manager.cc:4841] TS d1fe7c7fc4c145828049fd5b35af0997 (127.9.176.195:45309): tablet 5fac7e4ebab641b3ad2428a751310f59 (table test-table [id=205cb96a738640579e54ad4e9e5b12e6]) successfully deleted
I20260808 22:25:48.178611 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 4652
I20260808 22:25:48.188566 4843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:48.178825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.194:43029 (local address 127.0.0.1:56142)
0808 22:25:48.188340 (+ 9515us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:48.188434 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.194:43029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":9382}
I20260808 22:25:48.194291 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 4126
2026-08-08T22:25:48Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1024 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 22:25:48 dist-test-slave-ncp9 krb5kdc[4884](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 22:25:48 dist-test-slave-ncp9 krb5kdc[4884](info): set up 2 sockets
Aug 08 22:25:48 dist-test-slave-ncp9 krb5kdc[4884](info): commencing operation
krb5kdc: starting...
W20260808 22:25:49.316043 9923 mini_kdc.cc:121] Time spent starting KDC: real 1.098s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/krb5kdc/test-user.keytab.
Aug 08 22:25:49 dist-test-slave-ncp9 krb5kdc[4884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227949, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T22:25:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:25:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.176.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.176.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.176.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab.
Entry for principal kudu/127.9.176.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab.
I20260808 22:25:49.437306 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:34929
--webserver_interface=127.9.176.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab
--principal=kudu/127.9.176.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:39759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.176.254:34929
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:25:49.555305 4903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:25:49.555563 4903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:25:49.555619 4903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:25:49.559536 4903 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:25:49.559587 4903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:25:49.559613 4903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:25:49.559633 4903 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:25:49.559654 4903 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:25:49.564718 4903 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:39759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.176.254:34929
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab
--principal=kudu/127.9.176.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:34929
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.176.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4903
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:25:49.565686 4903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:25:49.566634 4903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:25:49.574669 4917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:25:49.574769 4903 server_base.cc:1034] running on GCE node
W20260808 22:25:49.574705 4910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:25:49.574714 4914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:25:49.575570 4903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:25:49.576579 4903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:25:49.577729 4903 hybrid_clock.cc:648] HybridClock initialized: now 1786227949577714 us; error 29 us; skew 500 ppm
Aug 08 22:25:49 dist-test-slave-ncp9 krb5kdc[4884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227949, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:25:49.580273 4903 init.cc:377] Logged in from keytab as kudu/127.9.176.254@KRBTEST.COM (short username kudu)
I20260808 22:25:49.581106 4903 webserver.cc:460] Webserver started at http://127.9.176.254:38181/ using document root <none> and password file <none>
I20260808 22:25:49.581601 4903 fs_manager.cc:359] Metadata directory not provided
I20260808 22:25:49.581667 4903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:25:49.581861 4903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:25:49.583807 4903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/minicluster-data/master-0/data/instance:
uuid: "b3daf1d5f19b4c988eb9165d1378c100"
format_stamp: "Formatted at 2026-08-08 22:25:49 on dist-test-slave-ncp9"
server_key: "069c1e225cb6651e7301a14e28ae1cc1"
server_key_iv: "7bf72fb856b20f71d6de26eeb9b804fe"
server_key_version: "encryptionkey@0"
I20260808 22:25:49.584335 4903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/minicluster-data/master-0/wal/instance:
uuid: "b3daf1d5f19b4c988eb9165d1378c100"
format_stamp: "Formatted at 2026-08-08 22:25:49 on dist-test-slave-ncp9"
server_key: "069c1e225cb6651e7301a14e28ae1cc1"
server_key_iv: "7bf72fb856b20f71d6de26eeb9b804fe"
server_key_version: "encryptionkey@0"
I20260808 22:25:49.587798 4903 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260808 22:25:49.590029 4928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:25:49.591236 4903 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 22:25:49.591363 4903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/minicluster-data/master-0/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/minicluster-data/master-0/wal
uuid: "b3daf1d5f19b4c988eb9165d1378c100"
format_stamp: "Formatted at 2026-08-08 22:25:49 on dist-test-slave-ncp9"
server_key: "069c1e225cb6651e7301a14e28ae1cc1"
server_key_iv: "7bf72fb856b20f71d6de26eeb9b804fe"
server_key_version: "encryptionkey@0"
I20260808 22:25:49.591459 4903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:25:49.612011 4903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:25:49.615005 4903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:25:49.615187 4903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:25:49.621842 4903 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.254:34929
I20260808 22:25:49.621887 4997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.254:34929 every 8 connection(s)
I20260808 22:25:49.622771 4903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
I20260808 22:25:49.624011 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 4903
I20260808 22:25:49.624135 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/minicluster-data/master-0/wal/instance
I20260808 22:25:49.624406 9923 external_mini_cluster.cc:1428] Setting key 2cb63408769c4f34592b8b64028436eb
I20260808 22:25:49.625697 4999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:25:49.632249 4999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3daf1d5f19b4c988eb9165d1378c100: Bootstrap starting.
Aug 08 22:25:49 dist-test-slave-ncp9 krb5kdc[4884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227949, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
I20260808 22:25:49.634544 4999 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3daf1d5f19b4c988eb9165d1378c100: Neither blocks nor log segments found. Creating new log.
I20260808 22:25:49.635416 4999 log.cc:824] T 00000000000000000000000000000000 P b3daf1d5f19b4c988eb9165d1378c100: Log is configured to *not* fsync() on all Append() calls
I20260808 22:25:49.637365 4999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3daf1d5f19b4c988eb9165d1378c100: No bootstrap required, opened a new log
I20260808 22:25:49.638896 5002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:49.625734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60130 (local address 127.9.176.254:34929)
0808 22:25:49.626163 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0808 22:25:49.626181 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0808 22:25:49.626222 (+ 41us) server_negotiation.cc:407] Connection header received
0808 22:25:49.627060 (+ 838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:25:49.627083 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:25:49.627486 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:25:49.627850 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:25:49.628517 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:49.630020 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:49.630916 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:49.631605 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:49.631760 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:49.634138 (+ 2378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:25:49.634143 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:25:49.634159 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:25:49.634189 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:25:49.636016 (+ 1827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:25:49.636429 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:25:49.636434 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:25:49.636438 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:25:49.636555 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:25:49.636810 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:25:49.636813 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:25:49.636815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:25:49.637231 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:25:49.637415 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0808 22:25:49.637698 (+ 283us) server_negotiation.cc:299] Negotiation successful
0808 22:25:49.637997 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":128,"threads_started":1}
I20260808 22:25:49.641394 4999 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3daf1d5f19b4c988eb9165d1378c100 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3daf1d5f19b4c988eb9165d1378c100" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 34929 } }
I20260808 22:25:49.641600 4999 raft_consensus.cc:374] T 00000000000000000000000000000000 P b3daf1d5f19b4c988eb9165d1378c100 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:25:49.641642 4999 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3daf1d5f19b4c988eb9165d1378c100 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3daf1d5f19b4c988eb9165d1378c100, State: Initialized, Role: FOLLOWER
I20260808 22:25:49.642007 4999 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3daf1d5f19b4c988eb9165d1378c100 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3daf1d5f19b4c988eb9165d1378c100" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 34929 } }
I20260808 22:25:49.642127 4999 raft_consensus.cc:388] T 00000000000000000000000000000000 P b3daf1d5f19b4c988eb9165d1378c100 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:25:49.642171 4999 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3daf1d5f19b4c988eb9165d1378c100 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:25:49.642254 4999 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3daf1d5f19b4c988eb9165d1378c100 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:25:49.643045 4999 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3daf1d5f19b4c988eb9165d1378c100 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3daf1d5f19b4c988eb9165d1378c100" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 34929 } }
I20260808 22:25:49.643324 4999 leader_election.cc:302] T 00000000000000000000000000000000 P b3daf1d5f19b4c988eb9165d1378c100 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b3daf1d5f19b4c988eb9165d1378c100; no voters:
I20260808 22:25:49.643568 4999 leader_election.cc:288] T 00000000000000000000000000000000 P b3daf1d5f19b4c988eb9165d1378c100 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:25:49.643704 5005 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3daf1d5f19b4c988eb9165d1378c100 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:25:49.643900 5005 raft_consensus.cc:686] T 00000000000000000000000000000000 P b3daf1d5f19b4c988eb9165d1378c100 [term 1 LEADER]: Becoming Leader. State: Replica: b3daf1d5f19b4c988eb9165d1378c100, State: Running, Role: LEADER
I20260808 22:25:49.644203 5005 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3daf1d5f19b4c988eb9165d1378c100 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3daf1d5f19b4c988eb9165d1378c100" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 34929 } }
I20260808 22:25:49.644670 4999 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3daf1d5f19b4c988eb9165d1378c100 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:25:49.645730 5007 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3daf1d5f19b4c988eb9165d1378c100 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b3daf1d5f19b4c988eb9165d1378c100" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3daf1d5f19b4c988eb9165d1378c100" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 34929 } } }
I20260808 22:25:49.645869 5007 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3daf1d5f19b4c988eb9165d1378c100 [sys.catalog]: This master's current role is: LEADER
I20260808 22:25:49.646239 5008 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3daf1d5f19b4c988eb9165d1378c100 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3daf1d5f19b4c988eb9165d1378c100. Latest consensus state: current_term: 1 leader_uuid: "b3daf1d5f19b4c988eb9165d1378c100" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3daf1d5f19b4c988eb9165d1378c100" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 34929 } } }
I20260808 22:25:49.646903 5008 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3daf1d5f19b4c988eb9165d1378c100 [sys.catalog]: This master's current role is: LEADER
I20260808 22:25:49.647107 5018 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:25:49.649744 5018 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:25:49.655131 5018 catalog_manager.cc:1347] Generated new cluster ID: 8029803fc23944cdb80b8416fe48849a
I20260808 22:25:49.655200 5018 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:25:49.688969 5018 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:25:49.690320 5018 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:25:49.696237 5018 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b3daf1d5f19b4c988eb9165d1378c100: Generated new TSK 0
I20260808 22:25:49.696993 5018 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.176.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.176.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.176.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.193.keytab.
Entry for principal kudu/127.9.176.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.193.keytab.
I20260808 22:25:49.725430 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.193:0
--local_ip_for_outbound_sockets=127.9.176.193
--webserver_interface=127.9.176.193
--webserver_port=0
--tserver_master_addrs=127.9.176.254:34929
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.193.keytab
--principal=kudu/127.9.176.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:39759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:25:49.836251 5040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:25:49.836550 5040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:25:49.836614 5040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:25:49.840588 5040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:25:49.840647 5040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:25:49.840723 5040 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.193
I20260808 22:25:49.845939 5040 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:39759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.193.keytab
--principal=kudu/127.9.176.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.176.193
--webserver_port=0
--tserver_master_addrs=127.9.176.254:34929
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5040
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.193
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:25:49.846948 5040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:25:49.847980 5040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:25:49.857036 5046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:25:49.857040 5047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:25:49.857115 5040 server_base.cc:1034] running on GCE node
W20260808 22:25:49.857057 5049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:25:49.857662 5040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:25:49.858244 5040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:25:49.859400 5040 hybrid_clock.cc:648] HybridClock initialized: now 1786227949859371 us; error 37 us; skew 500 ppm
Aug 08 22:25:49 dist-test-slave-ncp9 krb5kdc[4884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227949, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:25:49.865043 5040 init.cc:377] Logged in from keytab as kudu/127.9.176.193@KRBTEST.COM (short username kudu)
I20260808 22:25:49.865876 5040 webserver.cc:460] Webserver started at http://127.9.176.193:37377/ using document root <none> and password file <none>
I20260808 22:25:49.866456 5040 fs_manager.cc:359] Metadata directory not provided
I20260808 22:25:49.866521 5040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:25:49.866715 5040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:25:49.868700 5040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/minicluster-data/ts-0/data/instance:
uuid: "6222a8bf1ead47afa518785f50220c84"
format_stamp: "Formatted at 2026-08-08 22:25:49 on dist-test-slave-ncp9"
server_key: "bf84ac56e1e2124a54f5d25968f9b7b9"
server_key_iv: "e633dad445aff4b769add12cfb69ec20"
server_key_version: "encryptionkey@0"
I20260808 22:25:49.869285 5040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/minicluster-data/ts-0/wal/instance:
uuid: "6222a8bf1ead47afa518785f50220c84"
format_stamp: "Formatted at 2026-08-08 22:25:49 on dist-test-slave-ncp9"
server_key: "bf84ac56e1e2124a54f5d25968f9b7b9"
server_key_iv: "e633dad445aff4b769add12cfb69ec20"
server_key_version: "encryptionkey@0"
I20260808 22:25:49.872834 5040 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260808 22:25:49.875145 5059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:25:49.876348 5040 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260808 22:25:49.876444 5040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/minicluster-data/ts-0/wal
uuid: "6222a8bf1ead47afa518785f50220c84"
format_stamp: "Formatted at 2026-08-08 22:25:49 on dist-test-slave-ncp9"
server_key: "bf84ac56e1e2124a54f5d25968f9b7b9"
server_key_iv: "e633dad445aff4b769add12cfb69ec20"
server_key_version: "encryptionkey@0"
I20260808 22:25:49.876528 5040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:25:49.890552 5040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:25:49.893397 5040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:25:49.893540 5040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:25:49.894042 5040 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:25:49.894812 5040 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:25:49.894858 5040 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:25:49.894896 5040 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:25:49.894915 5040 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:25:49.903033 5040 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.193:33535
I20260808 22:25:49.903077 5228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.193:33535 every 8 connection(s)
I20260808 22:25:49.903851 5040 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
I20260808 22:25:49.911543 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 5040
I20260808 22:25:49.911661 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/minicluster-data/ts-0/wal/instance
I20260808 22:25:49.912001 9923 external_mini_cluster.cc:1428] Setting key 95ae867ccbc838607edff87342d39d93
Aug 08 22:25:49 dist-test-slave-ncp9 krb5kdc[4884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227949, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.193@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
I20260808 22:25:49.916982 5002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:49.905542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.193:40359 (local address 127.9.176.254:34929)
0808 22:25:49.905663 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0808 22:25:49.905666 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:25:49.906431 (+ 765us) server_negotiation.cc:407] Connection header received
0808 22:25:49.907758 (+ 1327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:25:49.907762 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:25:49.907820 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:25:49.907890 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:25:49.909041 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:49.909627 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:49.910736 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:49.910967 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:49.911063 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:49.913588 (+ 2525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:25:49.913592 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:25:49.913595 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:25:49.913623 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:25:49.915020 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:25:49.915526 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:25:49.915529 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:25:49.915530 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:25:49.915610 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:25:49.915918 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:25:49.915922 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:25:49.915923 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:25:49.916100 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:25:49.916186 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0808 22:25:49.916687 (+ 501us) server_negotiation.cc:299] Negotiation successful
0808 22:25:49.916810 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260808 22:25:49.917708 5232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:49.905863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:34929 (local address 127.9.176.193:40359)
0808 22:25:49.906290 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:49.906329 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 22:25:49.907553 (+ 1224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:25:49.908065 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:25:49.908072 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:25:49.908394 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:25:49.908856 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:49.908870 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:49.909748 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:49.909751 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:49.910617 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:49.910627 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:49.911110 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:49.911114 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:49.911269 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:49.911786 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:25:49.911807 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:25:49.913419 (+ 1612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:25:49.915137 (+ 1718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:25:49.915142 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:25:49.915153 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:25:49.915422 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:25:49.915694 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:25:49.915697 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:25:49.915699 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:25:49.915830 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:25:49.916199 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:25:49.916206 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:25:49.916430 (+ 224us) client_negotiation.cc:769] Sending connection context
0808 22:25:49.916608 (+ 178us) client_negotiation.cc:240] Negotiation successful
0808 22:25:49.916850 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":118,"threads_started":1}
I20260808 22:25:49.919016 5229 heartbeater.cc:347] Connected to a master server at 127.9.176.254:34929
I20260808 22:25:49.919226 5229 heartbeater.cc:464] Registering TS with master...
I20260808 22:25:49.919658 5229 heartbeater.cc:511] Master 127.9.176.254:34929 requested a full tablet report, sending...
I20260808 22:25:49.921110 4950 ts_manager.cc:194] Registered new tserver with Master: 6222a8bf1ead47afa518785f50220c84 (127.9.176.193:33535)
WARNING: no policy specified for kudu/127.9.176.194@KRBTEST.COM; defaulting to no policy
I20260808 22:25:49.922223 4950 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.176.193@KRBTEST.COM'} at 127.9.176.193:40359
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.176.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.176.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.194.keytab.
Entry for principal kudu/127.9.176.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.194.keytab.
I20260808 22:25:49.937232 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.194:0
--local_ip_for_outbound_sockets=127.9.176.194
--webserver_interface=127.9.176.194
--webserver_port=0
--tserver_master_addrs=127.9.176.254:34929
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.194.keytab
--principal=kudu/127.9.176.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:39759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:25:50.047497 5235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:25:50.047735 5235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:25:50.047803 5235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:25:50.051729 5235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:25:50.051783 5235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:25:50.051858 5235 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.194
I20260808 22:25:50.056980 5235 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:39759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.194.keytab
--principal=kudu/127.9.176.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.176.194
--webserver_port=0
--tserver_master_addrs=127.9.176.254:34929
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5235
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.194
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:25:50.057896 5235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:25:50.058820 5235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:25:50.068051 5250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:25:50.068059 5244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:25:50.068034 5242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:25:50.068360 5235 server_base.cc:1034] running on GCE node
I20260808 22:25:50.068753 5235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:25:50.069378 5235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:25:50.070535 5235 hybrid_clock.cc:648] HybridClock initialized: now 1786227950070512 us; error 43 us; skew 500 ppm
Aug 08 22:25:50 dist-test-slave-ncp9 krb5kdc[4884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227950, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:25:50.073208 5235 init.cc:377] Logged in from keytab as kudu/127.9.176.194@KRBTEST.COM (short username kudu)
I20260808 22:25:50.073978 5235 webserver.cc:460] Webserver started at http://127.9.176.194:36739/ using document root <none> and password file <none>
I20260808 22:25:50.074463 5235 fs_manager.cc:359] Metadata directory not provided
I20260808 22:25:50.074520 5235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:25:50.074678 5235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:25:50.076645 5235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/minicluster-data/ts-1/data/instance:
uuid: "01232e3f5c554a9ea472569fdf90778c"
format_stamp: "Formatted at 2026-08-08 22:25:50 on dist-test-slave-ncp9"
server_key: "c9d97da595bb0de16fe9436bb951957c"
server_key_iv: "94961bcd98ac4a6f009b0938abb0a5cc"
server_key_version: "encryptionkey@0"
I20260808 22:25:50.077201 5235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/minicluster-data/ts-1/wal/instance:
uuid: "01232e3f5c554a9ea472569fdf90778c"
format_stamp: "Formatted at 2026-08-08 22:25:50 on dist-test-slave-ncp9"
server_key: "c9d97da595bb0de16fe9436bb951957c"
server_key_iv: "94961bcd98ac4a6f009b0938abb0a5cc"
server_key_version: "encryptionkey@0"
I20260808 22:25:50.080682 5235 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260808 22:25:50.083011 5259 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:25:50.084131 5235 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 22:25:50.084226 5235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/minicluster-data/ts-1/wal
uuid: "01232e3f5c554a9ea472569fdf90778c"
format_stamp: "Formatted at 2026-08-08 22:25:50 on dist-test-slave-ncp9"
server_key: "c9d97da595bb0de16fe9436bb951957c"
server_key_iv: "94961bcd98ac4a6f009b0938abb0a5cc"
server_key_version: "encryptionkey@0"
I20260808 22:25:50.084322 5235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:25:50.102054 5235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:25:50.105099 5235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:25:50.105273 5235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:25:50.105829 5235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:25:50.106616 5235 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:25:50.106662 5235 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:25:50.106702 5235 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:25:50.106730 5235 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:25:50.115272 5235 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.194:46597
I20260808 22:25:50.115320 5428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.194:46597 every 8 connection(s)
I20260808 22:25:50.116112 5235 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
I20260808 22:25:50.123407 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 5235
I20260808 22:25:50.123538 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/minicluster-data/ts-1/wal/instance
I20260808 22:25:50.123821 9923 external_mini_cluster.cc:1428] Setting key e3f3578fbf9127cb45c36941937bbf56
Aug 08 22:25:50 dist-test-slave-ncp9 krb5kdc[4884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227950, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.194@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
I20260808 22:25:50.129129 5002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:50.117924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.194:49415 (local address 127.9.176.254:34929)
0808 22:25:50.118064 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0808 22:25:50.118068 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:25:50.118877 (+ 809us) server_negotiation.cc:407] Connection header received
0808 22:25:50.120196 (+ 1319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:25:50.120199 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:25:50.120257 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:25:50.120330 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:25:50.121528 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:50.122060 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:50.123212 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:50.123435 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:50.123526 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:50.125990 (+ 2464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:25:50.125993 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:25:50.125995 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:25:50.126026 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:25:50.127271 (+ 1245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:25:50.127734 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:25:50.127737 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:25:50.127738 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:25:50.127812 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:25:50.128114 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:25:50.128117 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:25:50.128119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:25:50.128288 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:25:50.128372 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0808 22:25:50.128875 (+ 503us) server_negotiation.cc:299] Negotiation successful
0808 22:25:50.129014 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 22:25:50.130091 5432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:50.118257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:34929 (local address 127.9.176.194:49415)
0808 22:25:50.118722 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:50.118764 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 22:25:50.119982 (+ 1218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:25:50.120506 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:25:50.120542 (+ 36us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:25:50.120876 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:25:50.121375 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:50.121389 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:50.122190 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:50.122193 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:50.123079 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:50.123088 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:50.123581 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:50.123585 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:50.123749 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:50.124333 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:25:50.124353 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:25:50.125836 (+ 1483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:25:50.127374 (+ 1538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:25:50.127379 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:25:50.127393 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:25:50.127641 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:25:50.127891 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:25:50.127894 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:25:50.127897 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:25:50.128026 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:25:50.128386 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:25:50.128393 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:25:50.128629 (+ 236us) client_negotiation.cc:769] Sending connection context
0808 22:25:50.128811 (+ 182us) client_negotiation.cc:240] Negotiation successful
0808 22:25:50.129142 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":108,"threads_started":1}
I20260808 22:25:50.131235 5430 heartbeater.cc:347] Connected to a master server at 127.9.176.254:34929
I20260808 22:25:50.131445 5430 heartbeater.cc:464] Registering TS with master...
I20260808 22:25:50.131894 5430 heartbeater.cc:511] Master 127.9.176.254:34929 requested a full tablet report, sending...
I20260808 22:25:50.132869 4950 ts_manager.cc:194] Registered new tserver with Master: 01232e3f5c554a9ea472569fdf90778c (127.9.176.194:46597)
I20260808 22:25:50.133427 4950 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.176.194@KRBTEST.COM'} at 127.9.176.194:49415
WARNING: no policy specified for kudu/127.9.176.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.176.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.176.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.195.keytab.
Entry for principal kudu/127.9.176.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.195.keytab.
I20260808 22:25:50.151419 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.195:0
--local_ip_for_outbound_sockets=127.9.176.195
--webserver_interface=127.9.176.195
--webserver_port=0
--tserver_master_addrs=127.9.176.254:34929
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.195.keytab
--principal=kudu/127.9.176.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:39759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:25:50.260947 5436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:25:50.261185 5436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:25:50.261227 5436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:25:50.265123 5436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:25:50.265172 5436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:25:50.265228 5436 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.195
I20260808 22:25:50.270327 5436 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:39759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.195.keytab
--principal=kudu/127.9.176.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.176.195
--webserver_port=0
--tserver_master_addrs=127.9.176.254:34929
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5436
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.195
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:25:50.271230 5436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:25:50.272152 5436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:25:50.281436 5449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:25:50.281414 5443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:25:50.281488 5436 server_base.cc:1034] running on GCE node
W20260808 22:25:50.281427 5444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:25:50.282006 5436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:25:50.282608 5436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:25:50.283762 5436 hybrid_clock.cc:648] HybridClock initialized: now 1786227950283747 us; error 25 us; skew 500 ppm
Aug 08 22:25:50 dist-test-slave-ncp9 krb5kdc[4884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227950, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:25:50.286356 5436 init.cc:377] Logged in from keytab as kudu/127.9.176.195@KRBTEST.COM (short username kudu)
I20260808 22:25:50.287192 5436 webserver.cc:460] Webserver started at http://127.9.176.195:42237/ using document root <none> and password file <none>
I20260808 22:25:50.287698 5436 fs_manager.cc:359] Metadata directory not provided
I20260808 22:25:50.287756 5436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:25:50.287933 5436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:25:50.289950 5436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/minicluster-data/ts-2/data/instance:
uuid: "d51ca6faf29e40459ca5af651aa412e5"
format_stamp: "Formatted at 2026-08-08 22:25:50 on dist-test-slave-ncp9"
server_key: "c7ac2594d28bb75ded029ce5a552f5f2"
server_key_iv: "2dc56203d4fe8ead6bd2f74cc0e16e5f"
server_key_version: "encryptionkey@0"
I20260808 22:25:50.290469 5436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/minicluster-data/ts-2/wal/instance:
uuid: "d51ca6faf29e40459ca5af651aa412e5"
format_stamp: "Formatted at 2026-08-08 22:25:50 on dist-test-slave-ncp9"
server_key: "c7ac2594d28bb75ded029ce5a552f5f2"
server_key_iv: "2dc56203d4fe8ead6bd2f74cc0e16e5f"
server_key_version: "encryptionkey@0"
I20260808 22:25:50.293946 5436 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260808 22:25:50.296286 5458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:25:50.297499 5436 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 22:25:50.297598 5436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/minicluster-data/ts-2/wal
uuid: "d51ca6faf29e40459ca5af651aa412e5"
format_stamp: "Formatted at 2026-08-08 22:25:50 on dist-test-slave-ncp9"
server_key: "c7ac2594d28bb75ded029ce5a552f5f2"
server_key_iv: "2dc56203d4fe8ead6bd2f74cc0e16e5f"
server_key_version: "encryptionkey@0"
I20260808 22:25:50.297695 5436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:25:50.328830 5436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:25:50.332002 5436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:25:50.332190 5436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:25:50.332787 5436 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:25:50.333636 5436 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:25:50.333683 5436 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:25:50.333724 5436 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:25:50.333752 5436 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:25:50.342404 5436 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.195:44793
I20260808 22:25:50.342439 5630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.195:44793 every 8 connection(s)
I20260808 22:25:50.343283 5436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
I20260808 22:25:50.347801 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 5436
I20260808 22:25:50.347941 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/minicluster-data/ts-2/wal/instance
I20260808 22:25:50.348208 9923 external_mini_cluster.cc:1428] Setting key ed860fbef8a19d77c728b6cf8f78dfd8
Aug 08 22:25:50 dist-test-slave-ncp9 krb5kdc[4884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227950, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.195@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
I20260808 22:25:50.356741 5002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:50.345106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.195:49147 (local address 127.9.176.254:34929)
0808 22:25:50.345239 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0808 22:25:50.345243 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:25:50.346165 (+ 922us) server_negotiation.cc:407] Connection header received
0808 22:25:50.347583 (+ 1418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:25:50.347587 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:25:50.347646 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:25:50.347741 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:25:50.349119 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:50.349750 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:50.351019 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:50.351245 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:50.351336 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:50.353670 (+ 2334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:25:50.353673 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:25:50.353675 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:25:50.353700 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:25:50.354958 (+ 1258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:25:50.355409 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:25:50.355411 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:25:50.355412 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:25:50.355476 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:25:50.355819 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:25:50.355821 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:25:50.355822 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:25:50.355978 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:25:50.356046 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0808 22:25:50.356533 (+ 487us) server_negotiation.cc:299] Negotiation successful
0808 22:25:50.356624 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 22:25:50.357600 5634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:50.345457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:34929 (local address 127.9.176.195:49147)
0808 22:25:50.345914 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:50.346064 (+ 150us) client_negotiation.cc:174] Beginning negotiation
0808 22:25:50.347374 (+ 1310us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:25:50.347899 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:25:50.347908 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:25:50.348353 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:25:50.348920 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:50.348953 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:50.349875 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:50.349878 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:50.350901 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:50.350912 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:50.351363 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:50.351365 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:50.351507 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:50.352044 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:25:50.352061 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:25:50.353522 (+ 1461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:25:50.355060 (+ 1538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:25:50.355065 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:25:50.355075 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:25:50.355332 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:25:50.355576 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:25:50.355579 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:25:50.355581 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:25:50.355739 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:25:50.356074 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:25:50.356079 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:25:50.356329 (+ 250us) client_negotiation.cc:769] Sending connection context
0808 22:25:50.356499 (+ 170us) client_negotiation.cc:240] Negotiation successful
0808 22:25:50.356755 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":109,"threads_started":1}
I20260808 22:25:50.358655 5631 heartbeater.cc:347] Connected to a master server at 127.9.176.254:34929
I20260808 22:25:50.358897 5631 heartbeater.cc:464] Registering TS with master...
I20260808 22:25:50.359390 5631 heartbeater.cc:511] Master 127.9.176.254:34929 requested a full tablet report, sending...
I20260808 22:25:50.360407 4950 ts_manager.cc:194] Registered new tserver with Master: d51ca6faf29e40459ca5af651aa412e5 (127.9.176.195:44793)
I20260808 22:25:50.361205 4950 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.176.195@KRBTEST.COM'} at 127.9.176.195:49147
I20260808 22:25:50.361937 9923 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 22:25:50.371093 5002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:50.363785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60132 (local address 127.9.176.254:34929)
0808 22:25:50.363886 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0808 22:25:50.363890 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:25:50.364076 (+ 186us) server_negotiation.cc:407] Connection header received
0808 22:25:50.364184 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:25:50.364187 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:25:50.364227 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:25:50.364289 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:25:50.365004 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:50.365526 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:50.366351 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:50.366595 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:50.366684 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:50.368232 (+ 1548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:25:50.368235 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:25:50.368238 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:25:50.368260 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:25:50.369384 (+ 1124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:25:50.369890 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:25:50.369893 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:25:50.369894 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:25:50.369970 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:25:50.370274 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:25:50.370277 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:25:50.370279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:25:50.370477 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:25:50.370612 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0808 22:25:50.370867 (+ 255us) server_negotiation.cc:299] Negotiation successful
0808 22:25:50.370954 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260808 22:25:50.373785 4950 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60132:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 22:25:50.376724 4950 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 22:25:50.390576 5652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:50.384604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.195:44793 (local address 127.0.0.1:47702)
0808 22:25:50.385034 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:50.385084 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0808 22:25:50.385218 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:25:50.385909 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:25:50.385915 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:25:50.385940 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:25:50.386074 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:50.386081 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:50.387468 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:50.387472 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:50.388881 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:50.388922 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:50.390010 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:50.390015 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:50.390127 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:50.390176 (+ 49us) client_negotiation.cc:769] Sending connection context
0808 22:25:50.390287 (+ 111us) client_negotiation.cc:240] Negotiation successful
0808 22:25:50.390412 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":78,"threads_started":1}
I20260808 22:25:50.390614 5653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:50.384765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47702 (local address 127.9.176.195:44793)
0808 22:25:50.385253 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0808 22:25:50.385265 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 22:25:50.385289 (+ 24us) server_negotiation.cc:407] Connection header received
0808 22:25:50.385362 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:25:50.385366 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:25:50.385565 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:25:50.385725 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:25:50.386205 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:50.387320 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:50.389043 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:50.389878 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:50.389980 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:50.390118 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0808 22:25:50.390347 (+ 229us) server_negotiation.cc:299] Negotiation successful
0808 22:25:50.390449 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":183,"threads_started":1}
I20260808 22:25:50.391049 5654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:50.384646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.194:46597 (local address 127.0.0.1:32874)
0808 22:25:50.385624 (+ 978us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:50.385640 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 22:25:50.385735 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:25:50.386897 (+ 1162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:25:50.386900 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:25:50.386932 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:25:50.387041 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:50.387047 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:50.388355 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:50.388358 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:50.389492 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:50.389503 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:50.390655 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:50.390659 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:50.390754 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:50.390777 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 22:25:50.390850 (+ 73us) client_negotiation.cc:240] Negotiation successful
0808 22:25:50.390920 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":914,"mutex_wait_us":25,"spinlock_wait_cycles":5248,"thread_start_us":110,"threads_started":1}
I20260808 22:25:50.391122 5655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:50.386013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32874 (local address 127.9.176.194:46597)
0808 22:25:50.386365 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0808 22:25:50.386378 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 22:25:50.386394 (+ 16us) server_negotiation.cc:407] Connection header received
0808 22:25:50.386455 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:25:50.386458 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:25:50.386583 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:25:50.386740 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:25:50.387146 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:50.388213 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:50.389625 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:50.390368 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:50.390457 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:50.390586 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0808 22:25:50.390896 (+ 310us) server_negotiation.cc:299] Negotiation successful
0808 22:25:50.390986 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":165,"threads_started":1}
I20260808 22:25:50.391794 5647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:50.385374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.193:33535 (local address 127.0.0.1:59208)
0808 22:25:50.386807 (+ 1433us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:50.386822 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 22:25:50.386939 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:25:50.387665 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:25:50.387667 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:25:50.387684 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:25:50.387765 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:50.387770 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:50.389231 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:50.389235 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:50.390029 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:50.390039 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:50.391328 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:50.391332 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:50.391441 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:50.391497 (+ 56us) client_negotiation.cc:769] Sending connection context
0808 22:25:50.391579 (+ 82us) client_negotiation.cc:240] Negotiation successful
0808 22:25:50.391652 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1367,"mutex_wait_us":1,"spinlock_wait_cycles":2944,"thread_start_us":106,"threads_started":1}
I20260808 22:25:50.391875 5658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:50.386453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59208 (local address 127.9.176.193:33535)
0808 22:25:50.386766 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0808 22:25:50.386782 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 22:25:50.386912 (+ 130us) server_negotiation.cc:407] Connection header received
0808 22:25:50.387128 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:25:50.387135 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:25:50.387300 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:25:50.387484 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:25:50.387880 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:50.389096 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:50.390182 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:50.390994 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:50.391109 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:50.391255 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0808 22:25:50.391630 (+ 375us) server_negotiation.cc:299] Negotiation successful
0808 22:25:50.391728 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":101,"threads_started":1}
I20260808 22:25:50.394202 5531 tablet_service.cc:1467] Processing CreateTablet for tablet 9a074a3cc7ec4563a740454ad22f655c (DEFAULT_TABLE table=test-table [id=5ea90cd42d5f4a20a23f8383370fe06d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:25:50.394447 5136 tablet_service.cc:1467] Processing CreateTablet for tablet 9a074a3cc7ec4563a740454ad22f655c (DEFAULT_TABLE table=test-table [id=5ea90cd42d5f4a20a23f8383370fe06d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:25:50.394582 5325 tablet_service.cc:1467] Processing CreateTablet for tablet 9a074a3cc7ec4563a740454ad22f655c (DEFAULT_TABLE table=test-table [id=5ea90cd42d5f4a20a23f8383370fe06d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:25:50.395364 5136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a074a3cc7ec4563a740454ad22f655c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:25:50.395409 5531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a074a3cc7ec4563a740454ad22f655c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:25:50.395426 5325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a074a3cc7ec4563a740454ad22f655c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:25:50.401950 5659 tablet_bootstrap.cc:492] T 9a074a3cc7ec4563a740454ad22f655c P 6222a8bf1ead47afa518785f50220c84: Bootstrap starting.
I20260808 22:25:50.403676 5660 tablet_bootstrap.cc:492] T 9a074a3cc7ec4563a740454ad22f655c P d51ca6faf29e40459ca5af651aa412e5: Bootstrap starting.
I20260808 22:25:50.403676 5661 tablet_bootstrap.cc:492] T 9a074a3cc7ec4563a740454ad22f655c P 01232e3f5c554a9ea472569fdf90778c: Bootstrap starting.
I20260808 22:25:50.404170 5659 tablet_bootstrap.cc:654] T 9a074a3cc7ec4563a740454ad22f655c P 6222a8bf1ead47afa518785f50220c84: Neither blocks nor log segments found. Creating new log.
I20260808 22:25:50.405292 5659 log.cc:824] T 9a074a3cc7ec4563a740454ad22f655c P 6222a8bf1ead47afa518785f50220c84: Log is configured to *not* fsync() on all Append() calls
I20260808 22:25:50.405920 5660 tablet_bootstrap.cc:654] T 9a074a3cc7ec4563a740454ad22f655c P d51ca6faf29e40459ca5af651aa412e5: Neither blocks nor log segments found. Creating new log.
I20260808 22:25:50.406368 5661 tablet_bootstrap.cc:654] T 9a074a3cc7ec4563a740454ad22f655c P 01232e3f5c554a9ea472569fdf90778c: Neither blocks nor log segments found. Creating new log.
I20260808 22:25:50.407224 5659 tablet_bootstrap.cc:492] T 9a074a3cc7ec4563a740454ad22f655c P 6222a8bf1ead47afa518785f50220c84: No bootstrap required, opened a new log
I20260808 22:25:50.407243 5660 log.cc:824] T 9a074a3cc7ec4563a740454ad22f655c P d51ca6faf29e40459ca5af651aa412e5: Log is configured to *not* fsync() on all Append() calls
I20260808 22:25:50.407243 5661 log.cc:824] T 9a074a3cc7ec4563a740454ad22f655c P 01232e3f5c554a9ea472569fdf90778c: Log is configured to *not* fsync() on all Append() calls
I20260808 22:25:50.407416 5659 ts_tablet_manager.cc:1397] T 9a074a3cc7ec4563a740454ad22f655c P 6222a8bf1ead47afa518785f50220c84: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 22:25:50.409271 5660 tablet_bootstrap.cc:492] T 9a074a3cc7ec4563a740454ad22f655c P d51ca6faf29e40459ca5af651aa412e5: No bootstrap required, opened a new log
I20260808 22:25:50.409466 5660 ts_tablet_manager.cc:1397] T 9a074a3cc7ec4563a740454ad22f655c P d51ca6faf29e40459ca5af651aa412e5: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260808 22:25:50.409710 5661 tablet_bootstrap.cc:492] T 9a074a3cc7ec4563a740454ad22f655c P 01232e3f5c554a9ea472569fdf90778c: No bootstrap required, opened a new log
I20260808 22:25:50.409889 5661 ts_tablet_manager.cc:1397] T 9a074a3cc7ec4563a740454ad22f655c P 01232e3f5c554a9ea472569fdf90778c: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260808 22:25:50.410652 5659 raft_consensus.cc:348] T 9a074a3cc7ec4563a740454ad22f655c P 6222a8bf1ead47afa518785f50220c84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d51ca6faf29e40459ca5af651aa412e5" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 44793 } } peers { permanent_uuid: "01232e3f5c554a9ea472569fdf90778c" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 46597 } } peers { permanent_uuid: "6222a8bf1ead47afa518785f50220c84" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33535 } }
I20260808 22:25:50.410871 5659 raft_consensus.cc:374] T 9a074a3cc7ec4563a740454ad22f655c P 6222a8bf1ead47afa518785f50220c84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:25:50.410940 5659 raft_consensus.cc:729] T 9a074a3cc7ec4563a740454ad22f655c P 6222a8bf1ead47afa518785f50220c84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6222a8bf1ead47afa518785f50220c84, State: Initialized, Role: FOLLOWER
I20260808 22:25:50.411356 5659 consensus_queue.cc:260] T 9a074a3cc7ec4563a740454ad22f655c P 6222a8bf1ead47afa518785f50220c84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d51ca6faf29e40459ca5af651aa412e5" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 44793 } } peers { permanent_uuid: "01232e3f5c554a9ea472569fdf90778c" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 46597 } } peers { permanent_uuid: "6222a8bf1ead47afa518785f50220c84" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33535 } }
I20260808 22:25:50.411943 5229 heartbeater.cc:503] Master 127.9.176.254:34929 was elected leader, sending a full tablet report...
I20260808 22:25:50.413229 5659 ts_tablet_manager.cc:1428] T 9a074a3cc7ec4563a740454ad22f655c P 6222a8bf1ead47afa518785f50220c84: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260808 22:25:50.413242 5660 raft_consensus.cc:348] T 9a074a3cc7ec4563a740454ad22f655c P d51ca6faf29e40459ca5af651aa412e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d51ca6faf29e40459ca5af651aa412e5" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 44793 } } peers { permanent_uuid: "01232e3f5c554a9ea472569fdf90778c" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 46597 } } peers { permanent_uuid: "6222a8bf1ead47afa518785f50220c84" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33535 } }
I20260808 22:25:50.413578 5660 raft_consensus.cc:374] T 9a074a3cc7ec4563a740454ad22f655c P d51ca6faf29e40459ca5af651aa412e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:25:50.413669 5660 raft_consensus.cc:729] T 9a074a3cc7ec4563a740454ad22f655c P d51ca6faf29e40459ca5af651aa412e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d51ca6faf29e40459ca5af651aa412e5, State: Initialized, Role: FOLLOWER
I20260808 22:25:50.413568 5661 raft_consensus.cc:348] T 9a074a3cc7ec4563a740454ad22f655c P 01232e3f5c554a9ea472569fdf90778c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d51ca6faf29e40459ca5af651aa412e5" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 44793 } } peers { permanent_uuid: "01232e3f5c554a9ea472569fdf90778c" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 46597 } } peers { permanent_uuid: "6222a8bf1ead47afa518785f50220c84" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33535 } }
I20260808 22:25:50.413856 5661 raft_consensus.cc:374] T 9a074a3cc7ec4563a740454ad22f655c P 01232e3f5c554a9ea472569fdf90778c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:25:50.413928 5661 raft_consensus.cc:729] T 9a074a3cc7ec4563a740454ad22f655c P 01232e3f5c554a9ea472569fdf90778c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01232e3f5c554a9ea472569fdf90778c, State: Initialized, Role: FOLLOWER
I20260808 22:25:50.414392 5660 consensus_queue.cc:260] T 9a074a3cc7ec4563a740454ad22f655c P d51ca6faf29e40459ca5af651aa412e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d51ca6faf29e40459ca5af651aa412e5" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 44793 } } peers { permanent_uuid: "01232e3f5c554a9ea472569fdf90778c" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 46597 } } peers { permanent_uuid: "6222a8bf1ead47afa518785f50220c84" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33535 } }
I20260808 22:25:50.414371 5661 consensus_queue.cc:260] T 9a074a3cc7ec4563a740454ad22f655c P 01232e3f5c554a9ea472569fdf90778c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d51ca6faf29e40459ca5af651aa412e5" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 44793 } } peers { permanent_uuid: "01232e3f5c554a9ea472569fdf90778c" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 46597 } } peers { permanent_uuid: "6222a8bf1ead47afa518785f50220c84" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33535 } }
I20260808 22:25:50.415030 5631 heartbeater.cc:503] Master 127.9.176.254:34929 was elected leader, sending a full tablet report...
I20260808 22:25:50.415341 5661 ts_tablet_manager.cc:1428] T 9a074a3cc7ec4563a740454ad22f655c P 01232e3f5c554a9ea472569fdf90778c: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260808 22:25:50.415984 5430 heartbeater.cc:503] Master 127.9.176.254:34929 was elected leader, sending a full tablet report...
I20260808 22:25:50.417208 5660 ts_tablet_manager.cc:1428] T 9a074a3cc7ec4563a740454ad22f655c P d51ca6faf29e40459ca5af651aa412e5: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260808 22:25:50.514871 5666 raft_consensus.cc:482] T 9a074a3cc7ec4563a740454ad22f655c P 6222a8bf1ead47afa518785f50220c84 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 22:25:50.515118 5666 raft_consensus.cc:504] T 9a074a3cc7ec4563a740454ad22f655c P 6222a8bf1ead47afa518785f50220c84 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d51ca6faf29e40459ca5af651aa412e5" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 44793 } } peers { permanent_uuid: "01232e3f5c554a9ea472569fdf90778c" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 46597 } } peers { permanent_uuid: "6222a8bf1ead47afa518785f50220c84" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33535 } }
I20260808 22:25:50.516255 5666 leader_election.cc:288] T 9a074a3cc7ec4563a740454ad22f655c P 6222a8bf1ead47afa518785f50220c84 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d51ca6faf29e40459ca5af651aa412e5 (127.9.176.195:44793), 01232e3f5c554a9ea472569fdf90778c (127.9.176.194:46597)
I20260808 22:25:50.520416 5655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:50.516735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.193:57603 (local address 127.9.176.194:46597)
0808 22:25:50.516866 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0808 22:25:50.516871 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:25:50.517026 (+ 155us) server_negotiation.cc:407] Connection header received
0808 22:25:50.517187 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:25:50.517191 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:25:50.517243 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:25:50.517315 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:25:50.517638 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:50.518176 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:50.519411 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:50.519824 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:50.519916 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:50.519971 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0808 22:25:50.520213 (+ 242us) server_negotiation.cc:299] Negotiation successful
0808 22:25:50.520282 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260808 22:25:50.520434 5653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:50.516677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.193:45109 (local address 127.9.176.195:44793)
0808 22:25:50.516868 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0808 22:25:50.516872 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:25:50.517179 (+ 307us) server_negotiation.cc:407] Connection header received
0808 22:25:50.517313 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:25:50.517317 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:25:50.517369 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:25:50.517500 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:25:50.517768 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:50.518280 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:50.519398 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:50.519848 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:50.519962 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:50.520014 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0808 22:25:50.520231 (+ 217us) server_negotiation.cc:299] Negotiation successful
0808 22:25:50.520309 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260808 22:25:50.520457 5672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:50.516607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.195:44793 (local address 127.9.176.193:45109)
0808 22:25:50.517064 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:50.517077 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 22:25:50.517210 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:25:50.517504 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:25:50.517506 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:25:50.517521 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:25:50.517634 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:50.517640 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:50.518390 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:50.518392 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:50.519277 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:50.519285 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:50.519963 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:50.519966 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:50.520114 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:50.520133 (+ 19us) client_negotiation.cc:769] Sending connection context
0808 22:25:50.520202 (+ 69us) client_negotiation.cc:240] Negotiation successful
0808 22:25:50.520284 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"mutex_wait_us":26,"thread_start_us":133,"threads_started":1}
I20260808 22:25:50.520455 5671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:50.516545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.194:46597 (local address 127.9.176.193:57603)
0808 22:25:50.516872 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:50.516918 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0808 22:25:50.517073 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:25:50.517338 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:25:50.517341 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:25:50.517359 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:25:50.517504 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:50.517513 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:50.518317 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:50.518320 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:50.519280 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:50.519290 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:50.519940 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:50.519944 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:50.520050 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:50.520115 (+ 65us) client_negotiation.cc:769] Sending connection context
0808 22:25:50.520174 (+ 59us) client_negotiation.cc:240] Negotiation successful
0808 22:25:50.520275 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"mutex_wait_us":110,"thread_start_us":82,"threads_started":1}
I20260808 22:25:50.521194 5561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a074a3cc7ec4563a740454ad22f655c" candidate_uuid: "6222a8bf1ead47afa518785f50220c84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d51ca6faf29e40459ca5af651aa412e5" is_pre_election: true
I20260808 22:25:50.521276 5357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a074a3cc7ec4563a740454ad22f655c" candidate_uuid: "6222a8bf1ead47afa518785f50220c84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01232e3f5c554a9ea472569fdf90778c" is_pre_election: true
I20260808 22:25:50.521498 5561 raft_consensus.cc:2454] T 9a074a3cc7ec4563a740454ad22f655c P d51ca6faf29e40459ca5af651aa412e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6222a8bf1ead47afa518785f50220c84 in term 0.
I20260808 22:25:50.521584 5357 raft_consensus.cc:2454] T 9a074a3cc7ec4563a740454ad22f655c P 01232e3f5c554a9ea472569fdf90778c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6222a8bf1ead47afa518785f50220c84 in term 0.
I20260808 22:25:50.521978 5063 leader_election.cc:302] T 9a074a3cc7ec4563a740454ad22f655c P 6222a8bf1ead47afa518785f50220c84 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 01232e3f5c554a9ea472569fdf90778c, 6222a8bf1ead47afa518785f50220c84, d51ca6faf29e40459ca5af651aa412e5; no voters:
I20260808 22:25:50.522279 5666 raft_consensus.cc:2781] T 9a074a3cc7ec4563a740454ad22f655c P 6222a8bf1ead47afa518785f50220c84 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 22:25:50.522357 5666 raft_consensus.cc:482] T 9a074a3cc7ec4563a740454ad22f655c P 6222a8bf1ead47afa518785f50220c84 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 22:25:50.522398 5666 raft_consensus.cc:3037] T 9a074a3cc7ec4563a740454ad22f655c P 6222a8bf1ead47afa518785f50220c84 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:25:50.523442 5666 raft_consensus.cc:504] T 9a074a3cc7ec4563a740454ad22f655c P 6222a8bf1ead47afa518785f50220c84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d51ca6faf29e40459ca5af651aa412e5" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 44793 } } peers { permanent_uuid: "01232e3f5c554a9ea472569fdf90778c" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 46597 } } peers { permanent_uuid: "6222a8bf1ead47afa518785f50220c84" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33535 } }
I20260808 22:25:50.523756 5666 leader_election.cc:288] T 9a074a3cc7ec4563a740454ad22f655c P 6222a8bf1ead47afa518785f50220c84 [CANDIDATE]: Term 1 election: Requested vote from peers d51ca6faf29e40459ca5af651aa412e5 (127.9.176.195:44793), 01232e3f5c554a9ea472569fdf90778c (127.9.176.194:46597)
I20260808 22:25:50.524083 5357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a074a3cc7ec4563a740454ad22f655c" candidate_uuid: "6222a8bf1ead47afa518785f50220c84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01232e3f5c554a9ea472569fdf90778c"
I20260808 22:25:50.524137 5561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a074a3cc7ec4563a740454ad22f655c" candidate_uuid: "6222a8bf1ead47afa518785f50220c84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d51ca6faf29e40459ca5af651aa412e5"
I20260808 22:25:50.524190 5357 raft_consensus.cc:3037] T 9a074a3cc7ec4563a740454ad22f655c P 01232e3f5c554a9ea472569fdf90778c [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:25:50.524255 5561 raft_consensus.cc:3037] T 9a074a3cc7ec4563a740454ad22f655c P d51ca6faf29e40459ca5af651aa412e5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:25:50.525269 5357 raft_consensus.cc:2454] T 9a074a3cc7ec4563a740454ad22f655c P 01232e3f5c554a9ea472569fdf90778c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6222a8bf1ead47afa518785f50220c84 in term 1.
I20260808 22:25:50.525290 5561 raft_consensus.cc:2454] T 9a074a3cc7ec4563a740454ad22f655c P d51ca6faf29e40459ca5af651aa412e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6222a8bf1ead47afa518785f50220c84 in term 1.
I20260808 22:25:50.525574 5061 leader_election.cc:302] T 9a074a3cc7ec4563a740454ad22f655c P 6222a8bf1ead47afa518785f50220c84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 01232e3f5c554a9ea472569fdf90778c, 6222a8bf1ead47afa518785f50220c84, d51ca6faf29e40459ca5af651aa412e5; no voters:
I20260808 22:25:50.525741 5666 raft_consensus.cc:2781] T 9a074a3cc7ec4563a740454ad22f655c P 6222a8bf1ead47afa518785f50220c84 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:25:50.525947 5666 raft_consensus.cc:686] T 9a074a3cc7ec4563a740454ad22f655c P 6222a8bf1ead47afa518785f50220c84 [term 1 LEADER]: Becoming Leader. State: Replica: 6222a8bf1ead47afa518785f50220c84, State: Running, Role: LEADER
I20260808 22:25:50.526288 5666 consensus_queue.cc:237] T 9a074a3cc7ec4563a740454ad22f655c P 6222a8bf1ead47afa518785f50220c84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d51ca6faf29e40459ca5af651aa412e5" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 44793 } } peers { permanent_uuid: "01232e3f5c554a9ea472569fdf90778c" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 46597 } } peers { permanent_uuid: "6222a8bf1ead47afa518785f50220c84" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33535 } }
I20260808 22:25:50.529258 4949 catalog_manager.cc:5495] T 9a074a3cc7ec4563a740454ad22f655c P 6222a8bf1ead47afa518785f50220c84 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6222a8bf1ead47afa518785f50220c84 (127.9.176.193). New cstate: current_term: 1 leader_uuid: "6222a8bf1ead47afa518785f50220c84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d51ca6faf29e40459ca5af651aa412e5" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 44793 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01232e3f5c554a9ea472569fdf90778c" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 46597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6222a8bf1ead47afa518785f50220c84" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 33535 } health_report { overall_health: HEALTHY } } }
I20260808 22:25:50.542124 5658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:50.538974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59210 (local address 127.9.176.193:33535)
0808 22:25:50.539084 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0808 22:25:50.539097 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 22:25:50.539110 (+ 13us) server_negotiation.cc:407] Connection header received
0808 22:25:50.539220 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:25:50.539222 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:25:50.539271 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:25:50.539351 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:25:50.539523 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:50.540005 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:50.540878 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:50.541124 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:50.541191 (+ 67us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:50.541434 (+ 243us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:25:50.541814 (+ 380us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:25:50.541912 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0808 22:25:50.541980 (+ 68us) server_negotiation.cc:299] Negotiation successful
0808 22:25:50.542013 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260808 22:25:50.547727 5357 raft_consensus.cc:1261] T 9a074a3cc7ec4563a740454ad22f655c P 01232e3f5c554a9ea472569fdf90778c [term 1 FOLLOWER]: Refusing update from remote peer 6222a8bf1ead47afa518785f50220c84: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:25:50.547727 5561 raft_consensus.cc:1261] T 9a074a3cc7ec4563a740454ad22f655c P d51ca6faf29e40459ca5af651aa412e5 [term 1 FOLLOWER]: Refusing update from remote peer 6222a8bf1ead47afa518785f50220c84: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:25:50.548408 5673 consensus_queue.cc:1035] T 9a074a3cc7ec4563a740454ad22f655c P 6222a8bf1ead47afa518785f50220c84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01232e3f5c554a9ea472569fdf90778c" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 46597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:25:50.548573 5666 consensus_queue.cc:1035] T 9a074a3cc7ec4563a740454ad22f655c P 6222a8bf1ead47afa518785f50220c84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d51ca6faf29e40459ca5af651aa412e5" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 44793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:25:50.555586 5677 mvcc.cc:205] Tried to move back new op lower bound from 7316389685436960768 to 7316389685358084096. Current Snapshot: MvccSnapshot[applied={T|T < 7316389685436960768}]
I20260808 22:25:50.556106 5680 mvcc.cc:205] Tried to move back new op lower bound from 7316389685436960768 to 7316389685358084096. Current Snapshot: MvccSnapshot[applied={T|T < 7316389685436960768}]
I20260808 22:25:50.563926 4949 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60132:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:25:50.564072 4949 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60132:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:25:50.566250 4949 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b3daf1d5f19b4c988eb9165d1378c100: Sending DeleteTablet for 3 replicas of tablet 9a074a3cc7ec4563a740454ad22f655c
I20260808 22:25:50.566931 5531 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a074a3cc7ec4563a740454ad22f655c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:25:50 UTC) from {username='kudu', principal='kudu/127.9.176.254@KRBTEST.COM'} at 127.0.0.1:47702
I20260808 22:25:50.567026 5136 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a074a3cc7ec4563a740454ad22f655c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:25:50 UTC) from {username='kudu', principal='kudu/127.9.176.254@KRBTEST.COM'} at 127.0.0.1:59208
I20260808 22:25:50.567058 5325 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a074a3cc7ec4563a740454ad22f655c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:25:50 UTC) from {username='kudu', principal='kudu/127.9.176.254@KRBTEST.COM'} at 127.0.0.1:32874
I20260808 22:25:50.567553 5695 tablet_replica.cc:330] T 9a074a3cc7ec4563a740454ad22f655c P d51ca6faf29e40459ca5af651aa412e5: stopping tablet replica
I20260808 22:25:50.567549 5696 tablet_replica.cc:330] T 9a074a3cc7ec4563a740454ad22f655c P 6222a8bf1ead47afa518785f50220c84: stopping tablet replica
I20260808 22:25:50.567848 5695 raft_consensus.cc:2229] T 9a074a3cc7ec4563a740454ad22f655c P d51ca6faf29e40459ca5af651aa412e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:25:50.567848 5696 raft_consensus.cc:2229] T 9a074a3cc7ec4563a740454ad22f655c P 6222a8bf1ead47afa518785f50220c84 [term 1 LEADER]: Raft consensus shutting down.
I20260808 22:25:50.568066 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 5040
I20260808 22:25:50.568202 5696 raft_consensus.cc:2258] T 9a074a3cc7ec4563a740454ad22f655c P 6222a8bf1ead47afa518785f50220c84 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:25:50.568216 5695 raft_consensus.cc:2258] T 9a074a3cc7ec4563a740454ad22f655c P d51ca6faf29e40459ca5af651aa412e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:25:50.568527 5697 tablet_replica.cc:330] T 9a074a3cc7ec4563a740454ad22f655c P 01232e3f5c554a9ea472569fdf90778c: stopping tablet replica
I20260808 22:25:50.568811 5697 raft_consensus.cc:2229] T 9a074a3cc7ec4563a740454ad22f655c P 01232e3f5c554a9ea472569fdf90778c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:25:50.569113 5697 raft_consensus.cc:2258] T 9a074a3cc7ec4563a740454ad22f655c P 01232e3f5c554a9ea472569fdf90778c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:25:50.572343 5697 ts_tablet_manager.cc:1905] T 9a074a3cc7ec4563a740454ad22f655c P 01232e3f5c554a9ea472569fdf90778c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:25:50.572786 5695 ts_tablet_manager.cc:1905] T 9a074a3cc7ec4563a740454ad22f655c P d51ca6faf29e40459ca5af651aa412e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:25:50.575171 5697 ts_tablet_manager.cc:1918] T 9a074a3cc7ec4563a740454ad22f655c P 01232e3f5c554a9ea472569fdf90778c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 22:25:50.575278 5697 log.cc:1196] T 9a074a3cc7ec4563a740454ad22f655c P 01232e3f5c554a9ea472569fdf90778c: Deleting WAL directory at /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/minicluster-data/ts-1/wal/wals/9a074a3cc7ec4563a740454ad22f655c
I20260808 22:25:50.575381 5695 ts_tablet_manager.cc:1918] T 9a074a3cc7ec4563a740454ad22f655c P d51ca6faf29e40459ca5af651aa412e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 22:25:50.575500 5695 log.cc:1196] T 9a074a3cc7ec4563a740454ad22f655c P d51ca6faf29e40459ca5af651aa412e5: Deleting WAL directory at /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786227706282457-9923-0/minicluster-data/ts-2/wal/wals/9a074a3cc7ec4563a740454ad22f655c
I20260808 22:25:50.575680 5697 ts_tablet_manager.cc:1939] T 9a074a3cc7ec4563a740454ad22f655c P 01232e3f5c554a9ea472569fdf90778c: Deleting consensus metadata
I20260808 22:25:50.575824 5695 ts_tablet_manager.cc:1939] T 9a074a3cc7ec4563a740454ad22f655c P d51ca6faf29e40459ca5af651aa412e5: Deleting consensus metadata
I20260808 22:25:50.576922 4931 catalog_manager.cc:4841] TS d51ca6faf29e40459ca5af651aa412e5 (127.9.176.195:44793): tablet 9a074a3cc7ec4563a740454ad22f655c (table test-table [id=5ea90cd42d5f4a20a23f8383370fe06d]) successfully deleted
I20260808 22:25:50.576974 4930 catalog_manager.cc:4841] TS 01232e3f5c554a9ea472569fdf90778c (127.9.176.194:46597): tablet 9a074a3cc7ec4563a740454ad22f655c (table test-table [id=5ea90cd42d5f4a20a23f8383370fe06d]) successfully deleted
W20260808 22:25:50.580883 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.176.193:33535 (error 108)
I20260808 22:25:50.581063 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 5235
I20260808 22:25:50.586772 5652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:50.586487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.193:33535 (local address 127.0.0.1:59218)
0808 22:25:50.586633 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:50.586697 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.193:33535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260808 22:25:50.586930 4930 catalog_manager.cc:4584] TS 6222a8bf1ead47afa518785f50220c84 (127.9.176.193:33535): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9a074a3cc7ec4563a740454ad22f655c: Network error: Client connection negotiation failed: client connection to 127.9.176.193:33535: connect: Connection refused (error 111)
I20260808 22:25:50.591362 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 5436
I20260808 22:25:50.606069 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 4903
2026-08-08T22:25:50Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2416 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-08T22:25:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:25:50Z Disabled control of system clock
I20260808 22:25:50.643069 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786227706282457-9923-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786227706282457-9923-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786227706282457-9923-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:43491
--webserver_interface=127.9.176.254
--webserver_port=0
--builtin_ntp_servers=127.9.176.212:34715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.176.254:43491
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 22:25:50.753826 5704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:25:50.754079 5704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:25:50.754112 5704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:25:50.757953 5704 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:25:50.758001 5704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:25:50.758015 5704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:25:50.758030 5704 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:25:50.758051 5704 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:25:50.762641 5704 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:34715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786227706282457-9923-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786227706282457-9923-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.176.254:43491
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:43491
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.176.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5704
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786227706282457-9923-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:25:50.763599 5704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:25:50.764866 5704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:25:50.772835 5715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:25:50.772862 5712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:25:50.772948 5711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:25:50.773154 5704 server_base.cc:1034] running on GCE node
I20260808 22:25:50.773676 5704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:25:50.774718 5704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:25:50.775878 5704 hybrid_clock.cc:648] HybridClock initialized: now 1786227950775863 us; error 25 us; skew 500 ppm
I20260808 22:25:50.777763 5704 webserver.cc:460] Webserver started at http://127.9.176.254:37349/ using document root <none> and password file <none>
I20260808 22:25:50.778306 5704 fs_manager.cc:359] Metadata directory not provided
I20260808 22:25:50.778373 5704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:25:50.778568 5704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:25:50.780540 5704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786227706282457-9923-0/minicluster-data/master-0/data/instance:
uuid: "d31e6c31e8c246efbf2a4a632d11e7a5"
format_stamp: "Formatted at 2026-08-08 22:25:50 on dist-test-slave-ncp9"
server_key: "8ba592ea0ae59a20e97aa76bd49f4620"
server_key_iv: "b09f0cef7f5317c4ccac7bb4bfb2034c"
server_key_version: "encryptionkey@0"
I20260808 22:25:50.781123 5704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786227706282457-9923-0/minicluster-data/master-0/wal/instance:
uuid: "d31e6c31e8c246efbf2a4a632d11e7a5"
format_stamp: "Formatted at 2026-08-08 22:25:50 on dist-test-slave-ncp9"
server_key: "8ba592ea0ae59a20e97aa76bd49f4620"
server_key_iv: "b09f0cef7f5317c4ccac7bb4bfb2034c"
server_key_version: "encryptionkey@0"
I20260808 22:25:50.784518 5704 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260808 22:25:50.786801 5723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:25:50.787949 5704 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 22:25:50.788074 5704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786227706282457-9923-0/minicluster-data/master-0/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786227706282457-9923-0/minicluster-data/master-0/wal
uuid: "d31e6c31e8c246efbf2a4a632d11e7a5"
format_stamp: "Formatted at 2026-08-08 22:25:50 on dist-test-slave-ncp9"
server_key: "8ba592ea0ae59a20e97aa76bd49f4620"
server_key_iv: "b09f0cef7f5317c4ccac7bb4bfb2034c"
server_key_version: "encryptionkey@0"
I20260808 22:25:50.788164 5704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786227706282457-9923-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786227706282457-9923-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786227706282457-9923-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:25:50.805302 5704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:25:50.805977 5704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:25:50.806149 5704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:25:50.812834 5704 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.254:43491
I20260808 22:25:50.812846 5801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.254:43491 every 8 connection(s)
I20260808 22:25:50.813796 5704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
I20260808 22:25:50.816797 5802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:25:50.818638 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 5704
I20260808 22:25:50.818769 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786227706282457-9923-0/minicluster-data/master-0/wal/instance
I20260808 22:25:50.819070 9923 external_mini_cluster.cc:1428] Setting key a18fb8c020cfb00ac3508d41feb56c0a
I20260808 22:25:50.823439 5802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d31e6c31e8c246efbf2a4a632d11e7a5: Bootstrap starting.
I20260808 22:25:50.825951 5802 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d31e6c31e8c246efbf2a4a632d11e7a5: Neither blocks nor log segments found. Creating new log.
I20260808 22:25:50.826761 5802 log.cc:824] T 00000000000000000000000000000000 P d31e6c31e8c246efbf2a4a632d11e7a5: Log is configured to *not* fsync() on all Append() calls
I20260808 22:25:50.828248 5808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:50.820437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52630 (local address 127.9.176.254:43491)
0808 22:25:50.821086 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0808 22:25:50.821102 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 22:25:50.821137 (+ 35us) server_negotiation.cc:407] Connection header received
0808 22:25:50.821829 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:25:50.821846 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:25:50.822249 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:25:50.822522 (+ 273us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:25:50.822850 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:50.823863 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:50.825068 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:50.825535 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:50.825650 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:50.826498 (+ 848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:25:50.826505 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:25:50.826520 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:25:50.826899 (+ 379us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:25:50.826967 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:25:50.827001 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:25:50.827075 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0808 22:25:50.827246 (+ 171us) server_negotiation.cc:299] Negotiation successful
0808 22:25:50.827520 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":109,"threads_started":1}
I20260808 22:25:50.828720 5802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d31e6c31e8c246efbf2a4a632d11e7a5: No bootstrap required, opened a new log
I20260808 22:25:50.832427 5802 raft_consensus.cc:348] T 00000000000000000000000000000000 P d31e6c31e8c246efbf2a4a632d11e7a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d31e6c31e8c246efbf2a4a632d11e7a5" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 43491 } }
I20260808 22:25:50.832638 5802 raft_consensus.cc:374] T 00000000000000000000000000000000 P d31e6c31e8c246efbf2a4a632d11e7a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:25:50.832695 5802 raft_consensus.cc:729] T 00000000000000000000000000000000 P d31e6c31e8c246efbf2a4a632d11e7a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d31e6c31e8c246efbf2a4a632d11e7a5, State: Initialized, Role: FOLLOWER
I20260808 22:25:50.833115 5802 consensus_queue.cc:260] T 00000000000000000000000000000000 P d31e6c31e8c246efbf2a4a632d11e7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d31e6c31e8c246efbf2a4a632d11e7a5" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 43491 } }
I20260808 22:25:50.833244 5802 raft_consensus.cc:388] T 00000000000000000000000000000000 P d31e6c31e8c246efbf2a4a632d11e7a5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:25:50.833308 5802 raft_consensus.cc:482] T 00000000000000000000000000000000 P d31e6c31e8c246efbf2a4a632d11e7a5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:25:50.833402 5802 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d31e6c31e8c246efbf2a4a632d11e7a5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:25:50.834273 5802 raft_consensus.cc:504] T 00000000000000000000000000000000 P d31e6c31e8c246efbf2a4a632d11e7a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d31e6c31e8c246efbf2a4a632d11e7a5" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 43491 } }
I20260808 22:25:50.834656 5802 leader_election.cc:302] T 00000000000000000000000000000000 P d31e6c31e8c246efbf2a4a632d11e7a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d31e6c31e8c246efbf2a4a632d11e7a5; no voters:
I20260808 22:25:50.834977 5802 leader_election.cc:288] T 00000000000000000000000000000000 P d31e6c31e8c246efbf2a4a632d11e7a5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:25:50.835047 5811 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d31e6c31e8c246efbf2a4a632d11e7a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:25:50.835265 5811 raft_consensus.cc:686] T 00000000000000000000000000000000 P d31e6c31e8c246efbf2a4a632d11e7a5 [term 1 LEADER]: Becoming Leader. State: Replica: d31e6c31e8c246efbf2a4a632d11e7a5, State: Running, Role: LEADER
I20260808 22:25:50.835636 5811 consensus_queue.cc:237] T 00000000000000000000000000000000 P d31e6c31e8c246efbf2a4a632d11e7a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d31e6c31e8c246efbf2a4a632d11e7a5" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 43491 } }
I20260808 22:25:50.836135 5802 sys_catalog.cc:564] T 00000000000000000000000000000000 P d31e6c31e8c246efbf2a4a632d11e7a5 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:25:50.837353 5812 sys_catalog.cc:455] T 00000000000000000000000000000000 P d31e6c31e8c246efbf2a4a632d11e7a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d31e6c31e8c246efbf2a4a632d11e7a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d31e6c31e8c246efbf2a4a632d11e7a5" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 43491 } } }
I20260808 22:25:50.837349 5813 sys_catalog.cc:455] T 00000000000000000000000000000000 P d31e6c31e8c246efbf2a4a632d11e7a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d31e6c31e8c246efbf2a4a632d11e7a5. Latest consensus state: current_term: 1 leader_uuid: "d31e6c31e8c246efbf2a4a632d11e7a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d31e6c31e8c246efbf2a4a632d11e7a5" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 43491 } } }
I20260808 22:25:50.837492 5812 sys_catalog.cc:458] T 00000000000000000000000000000000 P d31e6c31e8c246efbf2a4a632d11e7a5 [sys.catalog]: This master's current role is: LEADER
I20260808 22:25:50.837498 5813 sys_catalog.cc:458] T 00000000000000000000000000000000 P d31e6c31e8c246efbf2a4a632d11e7a5 [sys.catalog]: This master's current role is: LEADER
I20260808 22:25:50.837945 5825 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:25:50.841022 5825 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:25:50.846310 5825 catalog_manager.cc:1347] Generated new cluster ID: 227cc97089324cb1a4af0d17c9704e27
I20260808 22:25:50.846370 5825 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:25:50.861452 5825 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:25:50.862351 5825 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:25:50.868455 5825 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d31e6c31e8c246efbf2a4a632d11e7a5: Generated new TSK 0
I20260808 22:25:50.869190 5825 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 22:25:50.879685 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786227706282457-9923-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786227706282457-9923-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786227706282457-9923-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.193:0
--local_ip_for_outbound_sockets=127.9.176.193
--webserver_interface=127.9.176.193
--webserver_port=0
--tserver_master_addrs=127.9.176.254:43491
--builtin_ntp_servers=127.9.176.212:34715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 22:25:51.028046 5837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:25:51.028286 5837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:25:51.028323 5837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:25:51.032217 5837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:25:51.032262 5837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:25:51.032313 5837 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.193
I20260808 22:25:51.036909 5837 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:34715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786227706282457-9923-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786227706282457-9923-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.176.193
--webserver_port=0
--tserver_master_addrs=127.9.176.254:43491
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5837
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.193
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786227706282457-9923-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:25:51.037912 5837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:25:51.039228 5837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:25:51.048583 5848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:25:51.048671 5837 server_base.cc:1034] running on GCE node
W20260808 22:25:51.048600 5852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:25:51.048600 5849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:25:51.049314 5837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:25:51.049903 5837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:25:51.051087 5837 hybrid_clock.cc:648] HybridClock initialized: now 1786227951051067 us; error 32 us; skew 500 ppm
I20260808 22:25:51.052999 5837 webserver.cc:460] Webserver started at http://127.9.176.193:45829/ using document root <none> and password file <none>
I20260808 22:25:51.053547 5837 fs_manager.cc:359] Metadata directory not provided
I20260808 22:25:51.053615 5837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:25:51.053819 5837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:25:51.055822 5837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786227706282457-9923-0/minicluster-data/ts-0/data/instance:
uuid: "34d1e88fb7d0484095b76a5b9e0d4b10"
format_stamp: "Formatted at 2026-08-08 22:25:51 on dist-test-slave-ncp9"
server_key: "199450072918a25c457fe69c77a2d6c7"
server_key_iv: "baff28e380d85eba77ebfbff85e15658"
server_key_version: "encryptionkey@0"
I20260808 22:25:51.056358 5837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786227706282457-9923-0/minicluster-data/ts-0/wal/instance:
uuid: "34d1e88fb7d0484095b76a5b9e0d4b10"
format_stamp: "Formatted at 2026-08-08 22:25:51 on dist-test-slave-ncp9"
server_key: "199450072918a25c457fe69c77a2d6c7"
server_key_iv: "baff28e380d85eba77ebfbff85e15658"
server_key_version: "encryptionkey@0"
I20260808 22:25:51.059816 5837 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260808 22:25:51.062212 5862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:25:51.063463 5837 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 22:25:51.063593 5837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786227706282457-9923-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786227706282457-9923-0/minicluster-data/ts-0/wal
uuid: "34d1e88fb7d0484095b76a5b9e0d4b10"
format_stamp: "Formatted at 2026-08-08 22:25:51 on dist-test-slave-ncp9"
server_key: "199450072918a25c457fe69c77a2d6c7"
server_key_iv: "baff28e380d85eba77ebfbff85e15658"
server_key_version: "encryptionkey@0"
I20260808 22:25:51.063683 5837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786227706282457-9923-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786227706282457-9923-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786227706282457-9923-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:25:51.084009 5837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:25:51.084638 5837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:25:51.084801 5837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:25:51.085386 5837 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:25:51.086297 5837 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:25:51.086345 5837 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:25:51.086396 5837 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:25:51.086422 5837 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:25:51.094892 5837 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.193:38515
I20260808 22:25:51.094955 6039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.193:38515 every 8 connection(s)
I20260808 22:25:51.095743 5837 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
I20260808 22:25:51.097638 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 5837
I20260808 22:25:51.097754 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786227706282457-9923-0/minicluster-data/ts-0/wal/instance
I20260808 22:25:51.098055 9923 external_mini_cluster.cc:1428] Setting key 33be7a2d033288766f55ccb65d88fced
I20260808 22:25:51.100157 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786227706282457-9923-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786227706282457-9923-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786227706282457-9923-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.194:0
--local_ip_for_outbound_sockets=127.9.176.194
--webserver_interface=127.9.176.194
--webserver_port=0
--tserver_master_addrs=127.9.176.254:43491
--builtin_ntp_servers=127.9.176.212:34715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 22:25:51.106380 5808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:51.097633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.193:47881 (local address 127.9.176.254:43491)
0808 22:25:51.097793 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0808 22:25:51.097798 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:25:51.098880 (+ 1082us) server_negotiation.cc:407] Connection header received
0808 22:25:51.100497 (+ 1617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:25:51.100499 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:25:51.100549 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:25:51.100619 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:25:51.101758 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:51.102606 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:51.104303 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:51.104535 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:51.104617 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:51.105939 (+ 1322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:25:51.105942 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:25:51.105944 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:25:51.105957 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:25:51.105978 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:25:51.105986 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:25:51.106050 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0808 22:25:51.106212 (+ 162us) server_negotiation.cc:299] Negotiation successful
0808 22:25:51.106267 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 22:25:51.107537 6044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:51.098010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:43491 (local address 127.9.176.193:47881)
0808 22:25:51.098736 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:51.098791 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0808 22:25:51.100287 (+ 1496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:25:51.100816 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:25:51.100824 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:25:51.100953 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:25:51.101586 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:51.101605 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:51.102754 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:51.102758 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:51.104189 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:51.104201 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:51.104689 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:51.104693 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:51.104887 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:51.105617 (+ 730us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 22:25:51.105619 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:25:51.105678 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 22:25:51.105764 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:25:51.106070 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:25:51.106080 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:25:51.106086 (+ 6us) client_negotiation.cc:769] Sending connection context
0808 22:25:51.106218 (+ 132us) client_negotiation.cc:240] Negotiation successful
0808 22:25:51.106521 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":128,"threads_started":1}
I20260808 22:25:51.108707 6040 heartbeater.cc:347] Connected to a master server at 127.9.176.254:43491
I20260808 22:25:51.108970 6040 heartbeater.cc:464] Registering TS with master...
I20260808 22:25:51.109493 6040 heartbeater.cc:511] Master 127.9.176.254:43491 requested a full tablet report, sending...
I20260808 22:25:51.110915 5748 ts_manager.cc:194] Registered new tserver with Master: 34d1e88fb7d0484095b76a5b9e0d4b10 (127.9.176.193:38515)
I20260808 22:25:51.112071 5748 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.176.193:47881
W20260808 22:25:51.212023 6045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:25:51.212255 6045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:25:51.212289 6045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:25:51.216176 6045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:25:51.216223 6045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:25:51.216279 6045 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.194
I20260808 22:25:51.220825 6045 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:34715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786227706282457-9923-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786227706282457-9923-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.176.194
--webserver_port=0
--tserver_master_addrs=127.9.176.254:43491
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6045
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.194
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786227706282457-9923-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:25:51.221861 6045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:25:51.223250 6045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:25:51.232622 6059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:25:51.232595 6062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:25:51.232659 6045 server_base.cc:1034] running on GCE node
W20260808 22:25:51.232599 6053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:25:51.233256 6045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:25:51.233891 6045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:25:51.235061 6045 hybrid_clock.cc:648] HybridClock initialized: now 1786227951235037 us; error 35 us; skew 500 ppm
I20260808 22:25:51.237079 6045 webserver.cc:460] Webserver started at http://127.9.176.194:35567/ using document root <none> and password file <none>
I20260808 22:25:51.237696 6045 fs_manager.cc:359] Metadata directory not provided
I20260808 22:25:51.237777 6045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:25:51.237991 6045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:25:51.240157 6045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786227706282457-9923-0/minicluster-data/ts-1/data/instance:
uuid: "c071c51a2fd84fcabbc3561e6d884810"
format_stamp: "Formatted at 2026-08-08 22:25:51 on dist-test-slave-ncp9"
server_key: "c63eabf2b739b7fbd21fd0acf30ca48e"
server_key_iv: "78c0c365dce582173c3fb39351ff1473"
server_key_version: "encryptionkey@0"
I20260808 22:25:51.240716 6045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786227706282457-9923-0/minicluster-data/ts-1/wal/instance:
uuid: "c071c51a2fd84fcabbc3561e6d884810"
format_stamp: "Formatted at 2026-08-08 22:25:51 on dist-test-slave-ncp9"
server_key: "c63eabf2b739b7fbd21fd0acf30ca48e"
server_key_iv: "78c0c365dce582173c3fb39351ff1473"
server_key_version: "encryptionkey@0"
I20260808 22:25:51.244676 6045 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260808 22:25:51.247242 6069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:25:51.248411 6045 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 22:25:51.248553 6045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786227706282457-9923-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786227706282457-9923-0/minicluster-data/ts-1/wal
uuid: "c071c51a2fd84fcabbc3561e6d884810"
format_stamp: "Formatted at 2026-08-08 22:25:51 on dist-test-slave-ncp9"
server_key: "c63eabf2b739b7fbd21fd0acf30ca48e"
server_key_iv: "78c0c365dce582173c3fb39351ff1473"
server_key_version: "encryptionkey@0"
I20260808 22:25:51.248653 6045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786227706282457-9923-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786227706282457-9923-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786227706282457-9923-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:25:51.259370 6045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:25:51.259987 6045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:25:51.260146 6045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:25:51.260684 6045 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:25:51.261596 6045 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:25:51.261642 6045 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:25:51.261691 6045 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:25:51.261720 6045 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:25:51.270260 6045 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.194:40541
I20260808 22:25:51.270283 6247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.194:40541 every 8 connection(s)
I20260808 22:25:51.271159 6045 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
I20260808 22:25:51.278162 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 6045
I20260808 22:25:51.278293 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786227706282457-9923-0/minicluster-data/ts-1/wal/instance
I20260808 22:25:51.278666 9923 external_mini_cluster.cc:1428] Setting key ec1481d89d139dd1f835fa86d9268ea4
I20260808 22:25:51.280118 5808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:51.272969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.194:33277 (local address 127.9.176.254:43491)
0808 22:25:51.273099 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0808 22:25:51.273103 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:25:51.274095 (+ 992us) server_negotiation.cc:407] Connection header received
0808 22:25:51.275444 (+ 1349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:25:51.275448 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:25:51.275504 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:25:51.275590 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:25:51.276441 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:51.277018 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:51.278205 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:51.278532 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:51.278652 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:51.279668 (+ 1016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:25:51.279672 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:25:51.279675 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:25:51.279687 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:25:51.279708 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:25:51.279717 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:25:51.279775 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0808 22:25:51.279919 (+ 144us) server_negotiation.cc:299] Negotiation successful
0808 22:25:51.279979 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260808 22:25:51.280706 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786227706282457-9923-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786227706282457-9923-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786227706282457-9923-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.195:0
--local_ip_for_outbound_sockets=127.9.176.195
--webserver_interface=127.9.176.195
--webserver_port=0
--tserver_master_addrs=127.9.176.254:43491
--builtin_ntp_servers=127.9.176.212:34715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 22:25:51.280966 6250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:51.273318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:43491 (local address 127.9.176.194:33277)
0808 22:25:51.273952 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:51.273990 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0808 22:25:51.275243 (+ 1253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:25:51.275739 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:25:51.275745 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:25:51.275824 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:25:51.276292 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:51.276311 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:51.277129 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:51.277132 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:51.278051 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:51.278063 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:51.278680 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:51.278683 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:51.278847 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:51.279418 (+ 571us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 22:25:51.279419 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:25:51.279457 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 22:25:51.279517 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:25:51.279784 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:25:51.279793 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:25:51.279800 (+ 7us) client_negotiation.cc:769] Sending connection context
0808 22:25:51.279903 (+ 103us) client_negotiation.cc:240] Negotiation successful
0808 22:25:51.280200 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":134,"threads_started":1}
I20260808 22:25:51.283332 6248 heartbeater.cc:347] Connected to a master server at 127.9.176.254:43491
I20260808 22:25:51.283552 6248 heartbeater.cc:464] Registering TS with master...
I20260808 22:25:51.284013 6248 heartbeater.cc:511] Master 127.9.176.254:43491 requested a full tablet report, sending...
I20260808 22:25:51.285007 5748 ts_manager.cc:194] Registered new tserver with Master: c071c51a2fd84fcabbc3561e6d884810 (127.9.176.194:40541)
I20260808 22:25:51.285543 5748 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.176.194:33277
W20260808 22:25:51.393651 6252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:25:51.393922 6252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:25:51.393975 6252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:25:51.398007 6252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:25:51.398067 6252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:25:51.398144 6252 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.195
I20260808 22:25:51.402920 6252 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:34715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786227706282457-9923-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786227706282457-9923-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.176.195
--webserver_port=0
--tserver_master_addrs=127.9.176.254:43491
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6252
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.195
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786227706282457-9923-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:25:51.403981 6252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:25:51.405426 6252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:25:51.414839 6260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:25:51.414847 6264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:25:51.414958 6252 server_base.cc:1034] running on GCE node
W20260808 22:25:51.414847 6259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:25:51.415478 6252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:25:51.416126 6252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:25:51.417279 6252 hybrid_clock.cc:648] HybridClock initialized: now 1786227951417261 us; error 29 us; skew 500 ppm
I20260808 22:25:51.419049 6252 webserver.cc:460] Webserver started at http://127.9.176.195:38395/ using document root <none> and password file <none>
I20260808 22:25:51.419603 6252 fs_manager.cc:359] Metadata directory not provided
I20260808 22:25:51.419672 6252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:25:51.419878 6252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:25:51.421931 6252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786227706282457-9923-0/minicluster-data/ts-2/data/instance:
uuid: "356d51ab3cb54c2080e2e29166be6989"
format_stamp: "Formatted at 2026-08-08 22:25:51 on dist-test-slave-ncp9"
server_key: "f5098ddb03ac6cf1cfe12e28a5921306"
server_key_iv: "49511f32609e965a20bd4744c0eaf56a"
server_key_version: "encryptionkey@0"
I20260808 22:25:51.422470 6252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786227706282457-9923-0/minicluster-data/ts-2/wal/instance:
uuid: "356d51ab3cb54c2080e2e29166be6989"
format_stamp: "Formatted at 2026-08-08 22:25:51 on dist-test-slave-ncp9"
server_key: "f5098ddb03ac6cf1cfe12e28a5921306"
server_key_iv: "49511f32609e965a20bd4744c0eaf56a"
server_key_version: "encryptionkey@0"
I20260808 22:25:51.425935 6252 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260808 22:25:51.428221 6271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:25:51.429551 6252 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 22:25:51.429672 6252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786227706282457-9923-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786227706282457-9923-0/minicluster-data/ts-2/wal
uuid: "356d51ab3cb54c2080e2e29166be6989"
format_stamp: "Formatted at 2026-08-08 22:25:51 on dist-test-slave-ncp9"
server_key: "f5098ddb03ac6cf1cfe12e28a5921306"
server_key_iv: "49511f32609e965a20bd4744c0eaf56a"
server_key_version: "encryptionkey@0"
I20260808 22:25:51.429766 6252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786227706282457-9923-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786227706282457-9923-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786227706282457-9923-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:25:51.441790 6252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:25:51.442410 6252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:25:51.442586 6252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:25:51.443172 6252 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:25:51.444065 6252 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:25:51.444114 6252 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:25:51.444164 6252 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:25:51.444195 6252 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:25:51.453188 6252 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.195:37219
I20260808 22:25:51.453248 6439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.195:37219 every 8 connection(s)
I20260808 22:25:51.454053 6252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
I20260808 22:25:51.457656 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 6252
I20260808 22:25:51.457767 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786227706282457-9923-0/minicluster-data/ts-2/wal/instance
I20260808 22:25:51.458103 9923 external_mini_cluster.cc:1428] Setting key df23a7f1298646dbe5cb04028fb8392c
I20260808 22:25:51.463511 5808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:51.455812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.195:44113 (local address 127.9.176.254:43491)
0808 22:25:51.455948 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0808 22:25:51.455952 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:25:51.456996 (+ 1044us) server_negotiation.cc:407] Connection header received
0808 22:25:51.458719 (+ 1723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:25:51.458722 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:25:51.458775 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:25:51.458850 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:25:51.459911 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:51.460526 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:51.461747 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:51.461962 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:51.462038 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:51.463155 (+ 1117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:25:51.463158 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:25:51.463160 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:25:51.463172 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:25:51.463191 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:25:51.463199 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:25:51.463246 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0808 22:25:51.463357 (+ 111us) server_negotiation.cc:299] Negotiation successful
0808 22:25:51.463403 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 22:25:51.464342 6446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:51.456184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:43491 (local address 127.9.176.195:44113)
0808 22:25:51.456817 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:51.456857 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 22:25:51.458486 (+ 1629us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:25:51.459049 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:25:51.459058 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:25:51.459166 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:25:51.459777 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:51.459792 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:51.460638 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:51.460641 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:51.461639 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:51.461649 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:51.462124 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:51.462128 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:51.462312 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:51.462885 (+ 573us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 22:25:51.462886 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:25:51.462928 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 22:25:51.463003 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:25:51.463256 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:25:51.463261 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:25:51.463265 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 22:25:51.463353 (+ 88us) client_negotiation.cc:240] Negotiation successful
0808 22:25:51.463575 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":109,"threads_started":1}
I20260808 22:25:51.465375 6440 heartbeater.cc:347] Connected to a master server at 127.9.176.254:43491
I20260808 22:25:51.465618 6440 heartbeater.cc:464] Registering TS with master...
I20260808 22:25:51.466117 6440 heartbeater.cc:511] Master 127.9.176.254:43491 requested a full tablet report, sending...
I20260808 22:25:51.467113 5748 ts_manager.cc:194] Registered new tserver with Master: 356d51ab3cb54c2080e2e29166be6989 (127.9.176.195:37219)
I20260808 22:25:51.467662 5748 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.176.195:44113
I20260808 22:25:51.471647 9923 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 22:25:51.476823 5808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:51.473491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52642 (local address 127.9.176.254:43491)
0808 22:25:51.473625 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0808 22:25:51.473629 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:25:51.473753 (+ 124us) server_negotiation.cc:407] Connection header received
0808 22:25:51.473836 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:25:51.473838 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:25:51.473874 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:25:51.473932 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:25:51.474111 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:51.474617 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:51.475367 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:51.475601 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:51.475675 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:51.476415 (+ 740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:25:51.476418 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:25:51.476420 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:25:51.476431 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:25:51.476459 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:25:51.476467 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:25:51.476566 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0808 22:25:51.476656 (+ 90us) server_negotiation.cc:299] Negotiation successful
0808 22:25:51.476690 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 22:25:51.479442 5748 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52642:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 22:25:51.482374 5748 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 22:25:51.495342 6458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:51.489851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.194:40541 (local address 127.0.0.1:39344)
0808 22:25:51.490463 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:51.490508 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 22:25:51.490630 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:25:51.491401 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:25:51.491407 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:25:51.491429 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:25:51.491562 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:51.491570 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:51.492616 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:51.492620 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:51.493903 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:51.493913 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:51.494836 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:51.494840 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:51.494943 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:51.494980 (+ 37us) client_negotiation.cc:769] Sending connection context
0808 22:25:51.495082 (+ 102us) client_negotiation.cc:240] Negotiation successful
0808 22:25:51.495198 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":75,"threads_started":1}
I20260808 22:25:51.495378 6459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:51.489988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39344 (local address 127.9.176.194:40541)
0808 22:25:51.490306 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0808 22:25:51.490318 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 22:25:51.490767 (+ 449us) server_negotiation.cc:407] Connection header received
0808 22:25:51.490867 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:25:51.490874 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:25:51.491069 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:25:51.491218 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:25:51.491694 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:51.492491 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:51.494031 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:51.494727 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:51.494822 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:51.495008 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0808 22:25:51.495130 (+ 122us) server_negotiation.cc:299] Negotiation successful
0808 22:25:51.495231 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":102,"threads_started":1}
I20260808 22:25:51.495654 6463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:51.490586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51066 (local address 127.9.176.195:37219)
0808 22:25:51.490994 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0808 22:25:51.491006 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 22:25:51.491029 (+ 23us) server_negotiation.cc:407] Connection header received
0808 22:25:51.491104 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:25:51.491109 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:25:51.491289 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:25:51.491432 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:25:51.491912 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:51.492920 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:51.493935 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:51.494739 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:51.494831 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:51.495011 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0808 22:25:51.495411 (+ 400us) server_negotiation.cc:299] Negotiation successful
0808 22:25:51.495501 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":100,"threads_started":1}
I20260808 22:25:51.495733 6461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:51.489959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.195:37219 (local address 127.0.0.1:51066)
0808 22:25:51.490837 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:51.490849 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 22:25:51.490929 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:25:51.491584 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:25:51.491586 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:25:51.491614 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:25:51.491789 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:51.491797 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:51.493016 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:51.493019 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:51.493779 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:51.493790 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:51.495065 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:51.495069 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:51.495166 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:51.495189 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 22:25:51.495258 (+ 69us) client_negotiation.cc:240] Negotiation successful
0808 22:25:51.495321 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":820,"mutex_wait_us":1,"thread_start_us":94,"threads_started":1}
I20260808 22:25:51.496466 6457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:51.490550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.193:38515 (local address 127.0.0.1:53708)
0808 22:25:51.491504 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:51.491517 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 22:25:51.491646 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:25:51.492316 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:25:51.492318 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:25:51.492335 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:25:51.492410 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:51.492415 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:51.493831 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:51.493834 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:51.494906 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:51.494916 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:51.496153 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:51.496155 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:51.496217 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:51.496232 (+ 15us) client_negotiation.cc:769] Sending connection context
0808 22:25:51.496286 (+ 54us) client_negotiation.cc:240] Negotiation successful
0808 22:25:51.496351 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":856,"thread_start_us":98,"threads_started":1}
I20260808 22:25:51.496583 6464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:51.490762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53708 (local address 127.9.176.193:38515)
0808 22:25:51.491336 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0808 22:25:51.491349 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 22:25:51.491587 (+ 238us) server_negotiation.cc:407] Connection header received
0808 22:25:51.491846 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:25:51.491852 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:25:51.492026 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:25:51.492156 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:25:51.492528 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:51.493712 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:51.495037 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:51.495861 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:51.495934 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:51.496072 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0808 22:25:51.496344 (+ 272us) server_negotiation.cc:299] Negotiation successful
0808 22:25:51.496435 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":130,"threads_started":1}
I20260808 22:25:51.498649 6342 tablet_service.cc:1467] Processing CreateTablet for tablet fb6ecc00b0854f1fb9d173625427f2fd (DEFAULT_TABLE table=test-table [id=3d151c16070d4b158fb5f104f5dc3a81]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:25:51.498967 5936 tablet_service.cc:1467] Processing CreateTablet for tablet fb6ecc00b0854f1fb9d173625427f2fd (DEFAULT_TABLE table=test-table [id=3d151c16070d4b158fb5f104f5dc3a81]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:25:51.499527 6342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb6ecc00b0854f1fb9d173625427f2fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:25:51.499728 5936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb6ecc00b0854f1fb9d173625427f2fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:25:51.503028 6147 tablet_service.cc:1467] Processing CreateTablet for tablet fb6ecc00b0854f1fb9d173625427f2fd (DEFAULT_TABLE table=test-table [id=3d151c16070d4b158fb5f104f5dc3a81]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:25:51.504015 6147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb6ecc00b0854f1fb9d173625427f2fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:25:51.505609 6465 tablet_bootstrap.cc:492] T fb6ecc00b0854f1fb9d173625427f2fd P 356d51ab3cb54c2080e2e29166be6989: Bootstrap starting.
I20260808 22:25:51.506992 6468 tablet_bootstrap.cc:492] T fb6ecc00b0854f1fb9d173625427f2fd P 34d1e88fb7d0484095b76a5b9e0d4b10: Bootstrap starting.
I20260808 22:25:51.508692 6465 tablet_bootstrap.cc:654] T fb6ecc00b0854f1fb9d173625427f2fd P 356d51ab3cb54c2080e2e29166be6989: Neither blocks nor log segments found. Creating new log.
I20260808 22:25:51.509037 6468 tablet_bootstrap.cc:654] T fb6ecc00b0854f1fb9d173625427f2fd P 34d1e88fb7d0484095b76a5b9e0d4b10: Neither blocks nor log segments found. Creating new log.
I20260808 22:25:51.509600 6465 log.cc:824] T fb6ecc00b0854f1fb9d173625427f2fd P 356d51ab3cb54c2080e2e29166be6989: Log is configured to *not* fsync() on all Append() calls
I20260808 22:25:51.509810 6468 log.cc:824] T fb6ecc00b0854f1fb9d173625427f2fd P 34d1e88fb7d0484095b76a5b9e0d4b10: Log is configured to *not* fsync() on all Append() calls
I20260808 22:25:51.510016 6469 tablet_bootstrap.cc:492] T fb6ecc00b0854f1fb9d173625427f2fd P c071c51a2fd84fcabbc3561e6d884810: Bootstrap starting.
I20260808 22:25:51.511716 6465 tablet_bootstrap.cc:492] T fb6ecc00b0854f1fb9d173625427f2fd P 356d51ab3cb54c2080e2e29166be6989: No bootstrap required, opened a new log
I20260808 22:25:51.511759 6468 tablet_bootstrap.cc:492] T fb6ecc00b0854f1fb9d173625427f2fd P 34d1e88fb7d0484095b76a5b9e0d4b10: No bootstrap required, opened a new log
I20260808 22:25:51.511909 6465 ts_tablet_manager.cc:1397] T fb6ecc00b0854f1fb9d173625427f2fd P 356d51ab3cb54c2080e2e29166be6989: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 22:25:51.511938 6468 ts_tablet_manager.cc:1397] T fb6ecc00b0854f1fb9d173625427f2fd P 34d1e88fb7d0484095b76a5b9e0d4b10: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260808 22:25:51.512260 6469 tablet_bootstrap.cc:654] T fb6ecc00b0854f1fb9d173625427f2fd P c071c51a2fd84fcabbc3561e6d884810: Neither blocks nor log segments found. Creating new log.
I20260808 22:25:51.513067 6469 log.cc:824] T fb6ecc00b0854f1fb9d173625427f2fd P c071c51a2fd84fcabbc3561e6d884810: Log is configured to *not* fsync() on all Append() calls
I20260808 22:25:51.514972 6469 tablet_bootstrap.cc:492] T fb6ecc00b0854f1fb9d173625427f2fd P c071c51a2fd84fcabbc3561e6d884810: No bootstrap required, opened a new log
I20260808 22:25:51.515158 6469 ts_tablet_manager.cc:1397] T fb6ecc00b0854f1fb9d173625427f2fd P c071c51a2fd84fcabbc3561e6d884810: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260808 22:25:51.515251 6468 raft_consensus.cc:348] T fb6ecc00b0854f1fb9d173625427f2fd P 34d1e88fb7d0484095b76a5b9e0d4b10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c071c51a2fd84fcabbc3561e6d884810" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 40541 } } peers { permanent_uuid: "356d51ab3cb54c2080e2e29166be6989" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37219 } } peers { permanent_uuid: "34d1e88fb7d0484095b76a5b9e0d4b10" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38515 } }
I20260808 22:25:51.515462 6468 raft_consensus.cc:374] T fb6ecc00b0854f1fb9d173625427f2fd P 34d1e88fb7d0484095b76a5b9e0d4b10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:25:51.515520 6468 raft_consensus.cc:729] T fb6ecc00b0854f1fb9d173625427f2fd P 34d1e88fb7d0484095b76a5b9e0d4b10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34d1e88fb7d0484095b76a5b9e0d4b10, State: Initialized, Role: FOLLOWER
I20260808 22:25:51.515920 6468 consensus_queue.cc:260] T fb6ecc00b0854f1fb9d173625427f2fd P 34d1e88fb7d0484095b76a5b9e0d4b10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c071c51a2fd84fcabbc3561e6d884810" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 40541 } } peers { permanent_uuid: "356d51ab3cb54c2080e2e29166be6989" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37219 } } peers { permanent_uuid: "34d1e88fb7d0484095b76a5b9e0d4b10" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38515 } }
I20260808 22:25:51.515843 6465 raft_consensus.cc:348] T fb6ecc00b0854f1fb9d173625427f2fd P 356d51ab3cb54c2080e2e29166be6989 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c071c51a2fd84fcabbc3561e6d884810" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 40541 } } peers { permanent_uuid: "356d51ab3cb54c2080e2e29166be6989" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37219 } } peers { permanent_uuid: "34d1e88fb7d0484095b76a5b9e0d4b10" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38515 } }
I20260808 22:25:51.516073 6465 raft_consensus.cc:374] T fb6ecc00b0854f1fb9d173625427f2fd P 356d51ab3cb54c2080e2e29166be6989 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:25:51.516139 6465 raft_consensus.cc:729] T fb6ecc00b0854f1fb9d173625427f2fd P 356d51ab3cb54c2080e2e29166be6989 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 356d51ab3cb54c2080e2e29166be6989, State: Initialized, Role: FOLLOWER
I20260808 22:25:51.516582 6465 consensus_queue.cc:260] T fb6ecc00b0854f1fb9d173625427f2fd P 356d51ab3cb54c2080e2e29166be6989 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c071c51a2fd84fcabbc3561e6d884810" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 40541 } } peers { permanent_uuid: "356d51ab3cb54c2080e2e29166be6989" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37219 } } peers { permanent_uuid: "34d1e88fb7d0484095b76a5b9e0d4b10" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38515 } }
I20260808 22:25:51.516760 6040 heartbeater.cc:503] Master 127.9.176.254:43491 was elected leader, sending a full tablet report...
I20260808 22:25:51.516848 6468 ts_tablet_manager.cc:1428] T fb6ecc00b0854f1fb9d173625427f2fd P 34d1e88fb7d0484095b76a5b9e0d4b10: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260808 22:25:51.517496 6465 ts_tablet_manager.cc:1428] T fb6ecc00b0854f1fb9d173625427f2fd P 356d51ab3cb54c2080e2e29166be6989: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260808 22:25:51.517869 6440 heartbeater.cc:503] Master 127.9.176.254:43491 was elected leader, sending a full tablet report...
I20260808 22:25:51.518968 6469 raft_consensus.cc:348] T fb6ecc00b0854f1fb9d173625427f2fd P c071c51a2fd84fcabbc3561e6d884810 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c071c51a2fd84fcabbc3561e6d884810" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 40541 } } peers { permanent_uuid: "356d51ab3cb54c2080e2e29166be6989" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37219 } } peers { permanent_uuid: "34d1e88fb7d0484095b76a5b9e0d4b10" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38515 } }
I20260808 22:25:51.519207 6469 raft_consensus.cc:374] T fb6ecc00b0854f1fb9d173625427f2fd P c071c51a2fd84fcabbc3561e6d884810 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:25:51.519263 6469 raft_consensus.cc:729] T fb6ecc00b0854f1fb9d173625427f2fd P c071c51a2fd84fcabbc3561e6d884810 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c071c51a2fd84fcabbc3561e6d884810, State: Initialized, Role: FOLLOWER
I20260808 22:25:51.519688 6469 consensus_queue.cc:260] T fb6ecc00b0854f1fb9d173625427f2fd P c071c51a2fd84fcabbc3561e6d884810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c071c51a2fd84fcabbc3561e6d884810" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 40541 } } peers { permanent_uuid: "356d51ab3cb54c2080e2e29166be6989" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37219 } } peers { permanent_uuid: "34d1e88fb7d0484095b76a5b9e0d4b10" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38515 } }
I20260808 22:25:51.520538 6469 ts_tablet_manager.cc:1428] T fb6ecc00b0854f1fb9d173625427f2fd P c071c51a2fd84fcabbc3561e6d884810: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260808 22:25:51.520848 6248 heartbeater.cc:503] Master 127.9.176.254:43491 was elected leader, sending a full tablet report...
W20260808 22:25:51.522058 6249 tablet.cc:2369] T fb6ecc00b0854f1fb9d173625427f2fd P c071c51a2fd84fcabbc3561e6d884810: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 22:25:51.525086 6480 raft_consensus.cc:482] T fb6ecc00b0854f1fb9d173625427f2fd P 34d1e88fb7d0484095b76a5b9e0d4b10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 22:25:51.525223 6480 raft_consensus.cc:504] T fb6ecc00b0854f1fb9d173625427f2fd P 34d1e88fb7d0484095b76a5b9e0d4b10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c071c51a2fd84fcabbc3561e6d884810" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 40541 } } peers { permanent_uuid: "356d51ab3cb54c2080e2e29166be6989" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37219 } } peers { permanent_uuid: "34d1e88fb7d0484095b76a5b9e0d4b10" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38515 } }
I20260808 22:25:51.526106 6480 leader_election.cc:288] T fb6ecc00b0854f1fb9d173625427f2fd P 34d1e88fb7d0484095b76a5b9e0d4b10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c071c51a2fd84fcabbc3561e6d884810 (127.9.176.194:40541), 356d51ab3cb54c2080e2e29166be6989 (127.9.176.195:37219)
I20260808 22:25:51.530294 6044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:51.526299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.194:40541 (local address 127.9.176.193:37917)
0808 22:25:51.526395 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:51.526409 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 22:25:51.526484 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:25:51.526711 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:25:51.526714 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:25:51.526734 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:25:51.526847 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:51.526855 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:51.527816 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:51.527819 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:51.529058 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:51.529069 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:51.529911 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:51.529914 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:51.529996 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:51.530022 (+ 26us) client_negotiation.cc:769] Sending connection context
0808 22:25:51.530087 (+ 65us) client_negotiation.cc:240] Negotiation successful
0808 22:25:51.530169 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260808 22:25:51.530297 6459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:51.526367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.193:37917 (local address 127.9.176.194:40541)
0808 22:25:51.526494 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0808 22:25:51.526498 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:25:51.526511 (+ 13us) server_negotiation.cc:407] Connection header received
0808 22:25:51.526554 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:25:51.526556 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:25:51.526599 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:25:51.526667 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:25:51.526962 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:51.527699 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:51.529195 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:51.529802 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:51.529926 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:51.529989 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0808 22:25:51.530114 (+ 125us) server_negotiation.cc:299] Negotiation successful
0808 22:25:51.530186 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 22:25:51.530349 6490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:51.526299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.195:37219 (local address 127.9.176.193:50023)
0808 22:25:51.526849 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:51.526862 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 22:25:51.526951 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:25:51.527148 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:25:51.527150 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:25:51.527162 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:25:51.527258 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:51.527262 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:51.528092 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:51.528096 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:51.529233 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:51.529241 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:51.530021 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:51.530024 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:51.530109 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:51.530129 (+ 20us) client_negotiation.cc:769] Sending connection context
0808 22:25:51.530190 (+ 61us) client_negotiation.cc:240] Negotiation successful
0808 22:25:51.530244 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"mutex_wait_us":247,"thread_start_us":113,"threads_started":1}
I20260808 22:25:51.530385 6463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:51.526423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.193:50023 (local address 127.9.176.195:37219)
0808 22:25:51.526529 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0808 22:25:51.526542 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 22:25:51.526940 (+ 398us) server_negotiation.cc:407] Connection header received
0808 22:25:51.527022 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:25:51.527025 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:25:51.527069 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:25:51.527133 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:25:51.527335 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:51.527984 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:51.529334 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:51.529929 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:51.530044 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:51.530086 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0808 22:25:51.530210 (+ 124us) server_negotiation.cc:299] Negotiation successful
0808 22:25:51.530276 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260808 22:25:51.531035 6380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb6ecc00b0854f1fb9d173625427f2fd" candidate_uuid: "34d1e88fb7d0484095b76a5b9e0d4b10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "356d51ab3cb54c2080e2e29166be6989" is_pre_election: true
I20260808 22:25:51.531111 6174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb6ecc00b0854f1fb9d173625427f2fd" candidate_uuid: "34d1e88fb7d0484095b76a5b9e0d4b10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c071c51a2fd84fcabbc3561e6d884810" is_pre_election: true
I20260808 22:25:51.531287 6380 raft_consensus.cc:2454] T fb6ecc00b0854f1fb9d173625427f2fd P 356d51ab3cb54c2080e2e29166be6989 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34d1e88fb7d0484095b76a5b9e0d4b10 in term 0.
I20260808 22:25:51.531337 6174 raft_consensus.cc:2454] T fb6ecc00b0854f1fb9d173625427f2fd P c071c51a2fd84fcabbc3561e6d884810 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34d1e88fb7d0484095b76a5b9e0d4b10 in term 0.
I20260808 22:25:51.531744 5869 leader_election.cc:302] T fb6ecc00b0854f1fb9d173625427f2fd P 34d1e88fb7d0484095b76a5b9e0d4b10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34d1e88fb7d0484095b76a5b9e0d4b10, 356d51ab3cb54c2080e2e29166be6989; no voters:
I20260808 22:25:51.531992 6480 raft_consensus.cc:2781] T fb6ecc00b0854f1fb9d173625427f2fd P 34d1e88fb7d0484095b76a5b9e0d4b10 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 22:25:51.532069 6480 raft_consensus.cc:482] T fb6ecc00b0854f1fb9d173625427f2fd P 34d1e88fb7d0484095b76a5b9e0d4b10 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 22:25:51.532101 6480 raft_consensus.cc:3037] T fb6ecc00b0854f1fb9d173625427f2fd P 34d1e88fb7d0484095b76a5b9e0d4b10 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:25:51.533185 6480 raft_consensus.cc:504] T fb6ecc00b0854f1fb9d173625427f2fd P 34d1e88fb7d0484095b76a5b9e0d4b10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c071c51a2fd84fcabbc3561e6d884810" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 40541 } } peers { permanent_uuid: "356d51ab3cb54c2080e2e29166be6989" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37219 } } peers { permanent_uuid: "34d1e88fb7d0484095b76a5b9e0d4b10" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38515 } }
I20260808 22:25:51.533492 6480 leader_election.cc:288] T fb6ecc00b0854f1fb9d173625427f2fd P 34d1e88fb7d0484095b76a5b9e0d4b10 [CANDIDATE]: Term 1 election: Requested vote from peers c071c51a2fd84fcabbc3561e6d884810 (127.9.176.194:40541), 356d51ab3cb54c2080e2e29166be6989 (127.9.176.195:37219)
I20260808 22:25:51.533867 6174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb6ecc00b0854f1fb9d173625427f2fd" candidate_uuid: "34d1e88fb7d0484095b76a5b9e0d4b10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c071c51a2fd84fcabbc3561e6d884810"
I20260808 22:25:51.533969 6174 raft_consensus.cc:3037] T fb6ecc00b0854f1fb9d173625427f2fd P c071c51a2fd84fcabbc3561e6d884810 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:25:51.533957 6380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb6ecc00b0854f1fb9d173625427f2fd" candidate_uuid: "34d1e88fb7d0484095b76a5b9e0d4b10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "356d51ab3cb54c2080e2e29166be6989"
I20260808 22:25:51.534085 6380 raft_consensus.cc:3037] T fb6ecc00b0854f1fb9d173625427f2fd P 356d51ab3cb54c2080e2e29166be6989 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:25:51.534958 6174 raft_consensus.cc:2454] T fb6ecc00b0854f1fb9d173625427f2fd P c071c51a2fd84fcabbc3561e6d884810 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34d1e88fb7d0484095b76a5b9e0d4b10 in term 1.
I20260808 22:25:51.535189 6380 raft_consensus.cc:2454] T fb6ecc00b0854f1fb9d173625427f2fd P 356d51ab3cb54c2080e2e29166be6989 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34d1e88fb7d0484095b76a5b9e0d4b10 in term 1.
I20260808 22:25:51.535225 5867 leader_election.cc:302] T fb6ecc00b0854f1fb9d173625427f2fd P 34d1e88fb7d0484095b76a5b9e0d4b10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34d1e88fb7d0484095b76a5b9e0d4b10, c071c51a2fd84fcabbc3561e6d884810; no voters:
I20260808 22:25:51.535369 6480 raft_consensus.cc:2781] T fb6ecc00b0854f1fb9d173625427f2fd P 34d1e88fb7d0484095b76a5b9e0d4b10 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:25:51.535722 6480 raft_consensus.cc:686] T fb6ecc00b0854f1fb9d173625427f2fd P 34d1e88fb7d0484095b76a5b9e0d4b10 [term 1 LEADER]: Becoming Leader. State: Replica: 34d1e88fb7d0484095b76a5b9e0d4b10, State: Running, Role: LEADER
I20260808 22:25:51.536033 6480 consensus_queue.cc:237] T fb6ecc00b0854f1fb9d173625427f2fd P 34d1e88fb7d0484095b76a5b9e0d4b10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c071c51a2fd84fcabbc3561e6d884810" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 40541 } } peers { permanent_uuid: "356d51ab3cb54c2080e2e29166be6989" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37219 } } peers { permanent_uuid: "34d1e88fb7d0484095b76a5b9e0d4b10" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38515 } }
I20260808 22:25:51.539153 5748 catalog_manager.cc:5495] T fb6ecc00b0854f1fb9d173625427f2fd P 34d1e88fb7d0484095b76a5b9e0d4b10 reported cstate change: term changed from 0 to 1, leader changed from <none> to 34d1e88fb7d0484095b76a5b9e0d4b10 (127.9.176.193). New cstate: current_term: 1 leader_uuid: "34d1e88fb7d0484095b76a5b9e0d4b10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c071c51a2fd84fcabbc3561e6d884810" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 40541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "356d51ab3cb54c2080e2e29166be6989" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37219 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "34d1e88fb7d0484095b76a5b9e0d4b10" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 38515 } health_report { overall_health: HEALTHY } } }
I20260808 22:25:51.559286 6464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:51.555785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53722 (local address 127.9.176.193:38515)
0808 22:25:51.555951 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0808 22:25:51.555955 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:25:51.555971 (+ 16us) server_negotiation.cc:407] Connection header received
0808 22:25:51.556047 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:25:51.556049 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:25:51.556107 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:25:51.556201 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:25:51.556428 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:51.557001 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:51.557821 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:51.558060 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:51.558154 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:51.558415 (+ 261us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:25:51.558880 (+ 465us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:25:51.559023 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0808 22:25:51.559069 (+ 46us) server_negotiation.cc:299] Negotiation successful
0808 22:25:51.559108 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 22:25:51.565066 6174 raft_consensus.cc:1261] T fb6ecc00b0854f1fb9d173625427f2fd P c071c51a2fd84fcabbc3561e6d884810 [term 1 FOLLOWER]: Refusing update from remote peer 34d1e88fb7d0484095b76a5b9e0d4b10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:25:51.565066 6380 raft_consensus.cc:1261] T fb6ecc00b0854f1fb9d173625427f2fd P 356d51ab3cb54c2080e2e29166be6989 [term 1 FOLLOWER]: Refusing update from remote peer 34d1e88fb7d0484095b76a5b9e0d4b10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:25:51.565722 6491 consensus_queue.cc:1035] T fb6ecc00b0854f1fb9d173625427f2fd P 34d1e88fb7d0484095b76a5b9e0d4b10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c071c51a2fd84fcabbc3561e6d884810" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 40541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:25:51.565893 6480 consensus_queue.cc:1035] T fb6ecc00b0854f1fb9d173625427f2fd P 34d1e88fb7d0484095b76a5b9e0d4b10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "356d51ab3cb54c2080e2e29166be6989" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 37219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:25:51.573719 6498 mvcc.cc:205] Tried to move back new op lower bound from 7316389689603547136 to 7316389689494159360. Current Snapshot: MvccSnapshot[applied={T|T < 7316389689603547136}]
I20260808 22:25:51.575208 6499 mvcc.cc:205] Tried to move back new op lower bound from 7316389689603547136 to 7316389689494159360. Current Snapshot: MvccSnapshot[applied={T|T < 7316389689603547136}]
I20260808 22:25:51.582602 5748 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52642:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:25:51.582765 5748 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52642:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:25:51.585214 5748 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d31e6c31e8c246efbf2a4a632d11e7a5: Sending DeleteTablet for 3 replicas of tablet fb6ecc00b0854f1fb9d173625427f2fd
I20260808 22:25:51.586030 6147 tablet_service.cc:1514] Processing DeleteTablet for tablet fb6ecc00b0854f1fb9d173625427f2fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:25:51 UTC) from {username='slave'} at 127.0.0.1:39344
I20260808 22:25:51.586047 5936 tablet_service.cc:1514] Processing DeleteTablet for tablet fb6ecc00b0854f1fb9d173625427f2fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:25:51 UTC) from {username='slave'} at 127.0.0.1:53708
I20260808 22:25:51.586251 6342 tablet_service.cc:1514] Processing DeleteTablet for tablet fb6ecc00b0854f1fb9d173625427f2fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:25:51 UTC) from {username='slave'} at 127.0.0.1:51066
I20260808 22:25:51.586575 6512 tablet_replica.cc:330] T fb6ecc00b0854f1fb9d173625427f2fd P c071c51a2fd84fcabbc3561e6d884810: stopping tablet replica
I20260808 22:25:51.586700 6511 tablet_replica.cc:330] T fb6ecc00b0854f1fb9d173625427f2fd P 34d1e88fb7d0484095b76a5b9e0d4b10: stopping tablet replica
I20260808 22:25:51.586903 6512 raft_consensus.cc:2229] T fb6ecc00b0854f1fb9d173625427f2fd P c071c51a2fd84fcabbc3561e6d884810 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:25:51.586979 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 5837
I20260808 22:25:51.586994 6511 raft_consensus.cc:2229] T fb6ecc00b0854f1fb9d173625427f2fd P 34d1e88fb7d0484095b76a5b9e0d4b10 [term 1 LEADER]: Raft consensus shutting down.
I20260808 22:25:51.587635 6513 tablet_replica.cc:330] T fb6ecc00b0854f1fb9d173625427f2fd P 356d51ab3cb54c2080e2e29166be6989: stopping tablet replica
I20260808 22:25:51.587698 6512 raft_consensus.cc:2258] T fb6ecc00b0854f1fb9d173625427f2fd P c071c51a2fd84fcabbc3561e6d884810 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:25:51.587895 6513 raft_consensus.cc:2229] T fb6ecc00b0854f1fb9d173625427f2fd P 356d51ab3cb54c2080e2e29166be6989 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:25:51.588176 6513 raft_consensus.cc:2258] T fb6ecc00b0854f1fb9d173625427f2fd P 356d51ab3cb54c2080e2e29166be6989 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:25:51.590988 6512 ts_tablet_manager.cc:1905] T fb6ecc00b0854f1fb9d173625427f2fd P c071c51a2fd84fcabbc3561e6d884810: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:25:51.591537 6513 ts_tablet_manager.cc:1905] T fb6ecc00b0854f1fb9d173625427f2fd P 356d51ab3cb54c2080e2e29166be6989: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:25:51.593803 6513 ts_tablet_manager.cc:1918] T fb6ecc00b0854f1fb9d173625427f2fd P 356d51ab3cb54c2080e2e29166be6989: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 22:25:51.593894 6513 log.cc:1196] T fb6ecc00b0854f1fb9d173625427f2fd P 356d51ab3cb54c2080e2e29166be6989: Deleting WAL directory at /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786227706282457-9923-0/minicluster-data/ts-2/wal/wals/fb6ecc00b0854f1fb9d173625427f2fd
I20260808 22:25:51.594139 6513 ts_tablet_manager.cc:1939] T fb6ecc00b0854f1fb9d173625427f2fd P 356d51ab3cb54c2080e2e29166be6989: Deleting consensus metadata
I20260808 22:25:51.594141 6512 ts_tablet_manager.cc:1918] T fb6ecc00b0854f1fb9d173625427f2fd P c071c51a2fd84fcabbc3561e6d884810: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 22:25:51.594234 6512 log.cc:1196] T fb6ecc00b0854f1fb9d173625427f2fd P c071c51a2fd84fcabbc3561e6d884810: Deleting WAL directory at /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786227706282457-9923-0/minicluster-data/ts-1/wal/wals/fb6ecc00b0854f1fb9d173625427f2fd
I20260808 22:25:51.594519 6512 ts_tablet_manager.cc:1939] T fb6ecc00b0854f1fb9d173625427f2fd P c071c51a2fd84fcabbc3561e6d884810: Deleting consensus metadata
I20260808 22:25:51.595417 5729 catalog_manager.cc:4841] TS 356d51ab3cb54c2080e2e29166be6989 (127.9.176.195:37219): tablet fb6ecc00b0854f1fb9d173625427f2fd (table test-table [id=3d151c16070d4b158fb5f104f5dc3a81]) successfully deleted
I20260808 22:25:51.595450 5728 catalog_manager.cc:4841] TS c071c51a2fd84fcabbc3561e6d884810 (127.9.176.194:40541): tablet fb6ecc00b0854f1fb9d173625427f2fd (table test-table [id=3d151c16070d4b158fb5f104f5dc3a81]) successfully deleted
W20260808 22:25:51.600644 5727 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.176.193:38515 (error 108)
I20260808 22:25:51.600860 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 6045
I20260808 22:25:51.607523 6457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:51.607219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.193:38515 (local address 127.0.0.1:53726)
0808 22:25:51.607363 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:51.607433 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.193:38515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260808 22:25:51.607690 5727 catalog_manager.cc:4584] TS 34d1e88fb7d0484095b76a5b9e0d4b10 (127.9.176.193:38515): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fb6ecc00b0854f1fb9d173625427f2fd: Network error: Client connection negotiation failed: client connection to 127.9.176.193:38515: connect: Connection refused (error 111)
I20260808 22:25:51.612018 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 6252
I20260808 22:25:51.623641 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 5704
2026-08-08T22:25:51Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1016 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 22:25:51 dist-test-slave-ncp9 krb5kdc[6523](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 22:25:51 dist-test-slave-ncp9 krb5kdc[6523](info): set up 2 sockets
Aug 08 22:25:51 dist-test-slave-ncp9 krb5kdc[6523](info): commencing operation
krb5kdc: starting...
W20260808 22:25:52.741320 9923 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/krb5kdc/test-user.keytab.
Aug 08 22:25:52 dist-test-slave-ncp9 krb5kdc[6523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227952, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T22:25:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:25:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.176.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.176.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.176.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab.
Entry for principal kudu/127.9.176.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab.
I20260808 22:25:52.855773 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:43315
--webserver_interface=127.9.176.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab
--principal=kudu/127.9.176.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:34079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.176.254:43315
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:25:52.963419 6544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:25:52.963658 6544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:25:52.963698 6544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:25:52.967638 6544 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:25:52.967697 6544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:25:52.967716 6544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:25:52.967747 6544 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:25:52.967773 6544 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:25:52.973024 6544 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:34079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.176.254:43315
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab
--principal=kudu/127.9.176.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:43315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.176.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6544
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:25:52.974002 6544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:25:52.974983 6544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:25:52.983136 6555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:25:52.983186 6554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:25:52.983299 6544 server_base.cc:1034] running on GCE node
W20260808 22:25:52.983186 6557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:25:52.983995 6544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:25:52.985039 6544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:25:52.986218 6544 hybrid_clock.cc:648] HybridClock initialized: now 1786227952986199 us; error 28 us; skew 500 ppm
Aug 08 22:25:52 dist-test-slave-ncp9 krb5kdc[6523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227952, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:25:52.988865 6544 init.cc:377] Logged in from keytab as kudu/127.9.176.254@KRBTEST.COM (short username kudu)
I20260808 22:25:52.989682 6544 webserver.cc:460] Webserver started at http://127.9.176.254:46749/ using document root <none> and password file <none>
I20260808 22:25:52.990182 6544 fs_manager.cc:359] Metadata directory not provided
I20260808 22:25:52.990247 6544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:25:52.990430 6544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:25:52.992375 6544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/minicluster-data/master-0/data/instance:
uuid: "fd986986e209479fb3bbbf41c958f233"
format_stamp: "Formatted at 2026-08-08 22:25:52 on dist-test-slave-ncp9"
server_key: "7f3d7de4a8593dcc72e427bf7fed4d40"
server_key_iv: "42273e0fe04655937181ec87643759cd"
server_key_version: "encryptionkey@0"
I20260808 22:25:52.992949 6544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/minicluster-data/master-0/wal/instance:
uuid: "fd986986e209479fb3bbbf41c958f233"
format_stamp: "Formatted at 2026-08-08 22:25:52 on dist-test-slave-ncp9"
server_key: "7f3d7de4a8593dcc72e427bf7fed4d40"
server_key_iv: "42273e0fe04655937181ec87643759cd"
server_key_version: "encryptionkey@0"
I20260808 22:25:52.996404 6544 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260808 22:25:52.998675 6568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:25:52.999817 6544 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 22:25:53.000018 6544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/minicluster-data/master-0/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/minicluster-data/master-0/wal
uuid: "fd986986e209479fb3bbbf41c958f233"
format_stamp: "Formatted at 2026-08-08 22:25:52 on dist-test-slave-ncp9"
server_key: "7f3d7de4a8593dcc72e427bf7fed4d40"
server_key_iv: "42273e0fe04655937181ec87643759cd"
server_key_version: "encryptionkey@0"
I20260808 22:25:53.000130 6544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:25:53.026496 6544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:25:53.029227 6544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:25:53.029400 6544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:25:53.036223 6544 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.254:43315
I20260808 22:25:53.036247 6639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.254:43315 every 8 connection(s)
I20260808 22:25:53.037184 6544 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
I20260808 22:25:53.039808 6640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:25:53.041326 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 6544
I20260808 22:25:53.041458 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/minicluster-data/master-0/wal/instance
I20260808 22:25:53.041700 9923 external_mini_cluster.cc:1428] Setting key 551757ce827317e658ce0d9555c7676a
I20260808 22:25:53.046260 6640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd986986e209479fb3bbbf41c958f233: Bootstrap starting.
I20260808 22:25:53.048789 6640 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd986986e209479fb3bbbf41c958f233: Neither blocks nor log segments found. Creating new log.
I20260808 22:25:53.049649 6640 log.cc:824] T 00000000000000000000000000000000 P fd986986e209479fb3bbbf41c958f233: Log is configured to *not* fsync() on all Append() calls
Aug 08 22:25:53 dist-test-slave-ncp9 krb5kdc[6523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227952, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
I20260808 22:25:53.051527 6640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd986986e209479fb3bbbf41c958f233: No bootstrap required, opened a new log
I20260808 22:25:53.055956 6640 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd986986e209479fb3bbbf41c958f233 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd986986e209479fb3bbbf41c958f233" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 43315 } }
I20260808 22:25:53.056214 6640 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd986986e209479fb3bbbf41c958f233 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:25:53.056272 6640 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd986986e209479fb3bbbf41c958f233 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd986986e209479fb3bbbf41c958f233, State: Initialized, Role: FOLLOWER
I20260808 22:25:53.056399 6643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:53.043066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49066 (local address 127.9.176.254:43315)
0808 22:25:53.043487 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0808 22:25:53.043507 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0808 22:25:53.043548 (+ 41us) server_negotiation.cc:407] Connection header received
0808 22:25:53.044377 (+ 829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:25:53.044401 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:25:53.044809 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:25:53.045205 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:25:53.045825 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:53.047244 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:53.048093 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:53.048771 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:53.048963 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:53.051289 (+ 2326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:25:53.051295 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:25:53.051311 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:25:53.051342 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:25:53.053359 (+ 2017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:25:53.053796 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:25:53.053803 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:25:53.053809 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:25:53.053967 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:25:53.054237 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:25:53.054241 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:25:53.054243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:25:53.054734 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:25:53.054947 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0808 22:25:53.055147 (+ 200us) server_negotiation.cc:299] Negotiation successful
0808 22:25:53.055446 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":107,"threads_started":1}
I20260808 22:25:53.056788 6640 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd986986e209479fb3bbbf41c958f233 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd986986e209479fb3bbbf41c958f233" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 43315 } }
I20260808 22:25:53.056943 6640 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd986986e209479fb3bbbf41c958f233 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:25:53.057010 6640 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd986986e209479fb3bbbf41c958f233 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:25:53.057113 6640 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd986986e209479fb3bbbf41c958f233 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:25:53.057973 6640 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd986986e209479fb3bbbf41c958f233 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd986986e209479fb3bbbf41c958f233" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 43315 } }
I20260808 22:25:53.058293 6640 leader_election.cc:302] T 00000000000000000000000000000000 P fd986986e209479fb3bbbf41c958f233 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd986986e209479fb3bbbf41c958f233; no voters:
I20260808 22:25:53.058646 6640 leader_election.cc:288] T 00000000000000000000000000000000 P fd986986e209479fb3bbbf41c958f233 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:25:53.058733 6645 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd986986e209479fb3bbbf41c958f233 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:25:53.058940 6645 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd986986e209479fb3bbbf41c958f233 [term 1 LEADER]: Becoming Leader. State: Replica: fd986986e209479fb3bbbf41c958f233, State: Running, Role: LEADER
I20260808 22:25:53.059243 6645 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd986986e209479fb3bbbf41c958f233 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd986986e209479fb3bbbf41c958f233" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 43315 } }
I20260808 22:25:53.059792 6640 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd986986e209479fb3bbbf41c958f233 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:25:53.060784 6650 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd986986e209479fb3bbbf41c958f233 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd986986e209479fb3bbbf41c958f233. Latest consensus state: current_term: 1 leader_uuid: "fd986986e209479fb3bbbf41c958f233" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd986986e209479fb3bbbf41c958f233" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 43315 } } }
I20260808 22:25:53.060948 6650 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd986986e209479fb3bbbf41c958f233 [sys.catalog]: This master's current role is: LEADER
I20260808 22:25:53.061242 6648 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd986986e209479fb3bbbf41c958f233 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd986986e209479fb3bbbf41c958f233" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd986986e209479fb3bbbf41c958f233" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 43315 } } }
I20260808 22:25:53.061316 6664 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:25:53.061379 6648 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd986986e209479fb3bbbf41c958f233 [sys.catalog]: This master's current role is: LEADER
I20260808 22:25:53.064616 6664 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:25:53.070288 6664 catalog_manager.cc:1347] Generated new cluster ID: 4daf440074314f3592f0af94e5f08c4b
I20260808 22:25:53.070355 6664 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:25:53.088939 6664 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:25:53.089860 6664 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:25:53.095325 6664 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd986986e209479fb3bbbf41c958f233: Generated new TSK 0
I20260808 22:25:53.096100 6664 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.176.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.176.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.176.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.193.keytab.
Entry for principal kudu/127.9.176.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.193.keytab.
I20260808 22:25:53.125397 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.193:0
--local_ip_for_outbound_sockets=127.9.176.193
--webserver_interface=127.9.176.193
--webserver_port=0
--tserver_master_addrs=127.9.176.254:43315
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.193.keytab
--principal=kudu/127.9.176.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:34079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:25:53.234975 6687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:25:53.235232 6687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:25:53.235275 6687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:25:53.239249 6687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:25:53.239311 6687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:25:53.239367 6687 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.193
I20260808 22:25:53.244555 6687 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:34079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.193.keytab
--principal=kudu/127.9.176.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.176.193
--webserver_port=0
--tserver_master_addrs=127.9.176.254:43315
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6687
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.193
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:25:53.245599 6687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:25:53.246572 6687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:25:53.255807 6694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:25:53.255826 6696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:25:53.255888 6699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:25:53.256013 6687 server_base.cc:1034] running on GCE node
I20260808 22:25:53.256449 6687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:25:53.257047 6687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:25:53.258204 6687 hybrid_clock.cc:648] HybridClock initialized: now 1786227953258184 us; error 32 us; skew 500 ppm
Aug 08 22:25:53 dist-test-slave-ncp9 krb5kdc[6523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227953, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:25:53.260915 6687 init.cc:377] Logged in from keytab as kudu/127.9.176.193@KRBTEST.COM (short username kudu)
I20260808 22:25:53.261760 6687 webserver.cc:460] Webserver started at http://127.9.176.193:44255/ using document root <none> and password file <none>
I20260808 22:25:53.262288 6687 fs_manager.cc:359] Metadata directory not provided
I20260808 22:25:53.262354 6687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:25:53.262550 6687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:25:53.264526 6687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/minicluster-data/ts-0/data/instance:
uuid: "d448ff9563af42e3bf7aaac8f9856c0d"
format_stamp: "Formatted at 2026-08-08 22:25:53 on dist-test-slave-ncp9"
server_key: "360fd631917d4cfdb179188a071abab3"
server_key_iv: "5c6a54b9d224450154f5d481854de138"
server_key_version: "encryptionkey@0"
I20260808 22:25:53.265112 6687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/minicluster-data/ts-0/wal/instance:
uuid: "d448ff9563af42e3bf7aaac8f9856c0d"
format_stamp: "Formatted at 2026-08-08 22:25:53 on dist-test-slave-ncp9"
server_key: "360fd631917d4cfdb179188a071abab3"
server_key_iv: "5c6a54b9d224450154f5d481854de138"
server_key_version: "encryptionkey@0"
I20260808 22:25:53.268519 6687 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260808 22:25:53.270774 6709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:25:53.271862 6687 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 22:25:53.271956 6687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/minicluster-data/ts-0/wal
uuid: "d448ff9563af42e3bf7aaac8f9856c0d"
format_stamp: "Formatted at 2026-08-08 22:25:53 on dist-test-slave-ncp9"
server_key: "360fd631917d4cfdb179188a071abab3"
server_key_iv: "5c6a54b9d224450154f5d481854de138"
server_key_version: "encryptionkey@0"
I20260808 22:25:53.272033 6687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:25:53.289974 6687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:25:53.292485 6687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:25:53.292654 6687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:25:53.293172 6687 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:25:53.293915 6687 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:25:53.293960 6687 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:25:53.293998 6687 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:25:53.294018 6687 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:25:53.302464 6687 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.193:46441
I20260808 22:25:53.302489 6868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.193:46441 every 8 connection(s)
I20260808 22:25:53.303328 6687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
I20260808 22:25:53.311506 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 6687
I20260808 22:25:53.311640 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/minicluster-data/ts-0/wal/instance
I20260808 22:25:53.311942 9923 external_mini_cluster.cc:1428] Setting key 1c25fc1bbb5766d79b5332a02d309099
Aug 08 22:25:53 dist-test-slave-ncp9 krb5kdc[6523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227953, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.193@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
I20260808 22:25:53.316215 6643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:53.305043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.193:35535 (local address 127.9.176.254:43315)
0808 22:25:53.305174 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0808 22:25:53.305178 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:25:53.305949 (+ 771us) server_negotiation.cc:407] Connection header received
0808 22:25:53.307222 (+ 1273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:25:53.307225 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:25:53.307270 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:25:53.307339 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:25:53.308405 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:53.308978 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:53.310091 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:53.310320 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:53.310394 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:53.312755 (+ 2361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:25:53.312759 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:25:53.312761 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:25:53.312786 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:25:53.314224 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:25:53.314781 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:25:53.314783 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:25:53.314785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:25:53.314852 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:25:53.315171 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:25:53.315173 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:25:53.315174 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:25:53.315354 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:25:53.315451 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0808 22:25:53.315937 (+ 486us) server_negotiation.cc:299] Negotiation successful
0808 22:25:53.316061 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 22:25:53.317039 6873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:53.305381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:43315 (local address 127.9.176.193:35535)
0808 22:25:53.305803 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:53.305844 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 22:25:53.307026 (+ 1182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:25:53.307485 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:25:53.307491 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:25:53.307821 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:25:53.308289 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:53.308303 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:53.309073 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:53.309076 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:53.309965 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:53.309974 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:53.310428 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:53.310430 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:53.310565 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:53.311035 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:25:53.311051 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:25:53.312595 (+ 1544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:25:53.314340 (+ 1745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:25:53.314346 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:25:53.314361 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:25:53.314694 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:25:53.314920 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:25:53.314923 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:25:53.314925 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:25:53.315096 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:25:53.315457 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:25:53.315463 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:25:53.315724 (+ 261us) client_negotiation.cc:769] Sending connection context
0808 22:25:53.315877 (+ 153us) client_negotiation.cc:240] Negotiation successful
0808 22:25:53.316113 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":111,"threads_started":1}
I20260808 22:25:53.318214 6869 heartbeater.cc:347] Connected to a master server at 127.9.176.254:43315
I20260808 22:25:53.318437 6869 heartbeater.cc:464] Registering TS with master...
I20260808 22:25:53.318888 6869 heartbeater.cc:511] Master 127.9.176.254:43315 requested a full tablet report, sending...
I20260808 22:25:53.320269 6588 ts_manager.cc:194] Registered new tserver with Master: d448ff9563af42e3bf7aaac8f9856c0d (127.9.176.193:46441)
I20260808 22:25:53.321487 6588 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.176.193@KRBTEST.COM'} at 127.9.176.193:35535
WARNING: no policy specified for kudu/127.9.176.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.176.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.176.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.194.keytab.
Entry for principal kudu/127.9.176.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.194.keytab.
I20260808 22:25:53.337323 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.194:0
--local_ip_for_outbound_sockets=127.9.176.194
--webserver_interface=127.9.176.194
--webserver_port=0
--tserver_master_addrs=127.9.176.254:43315
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.194.keytab
--principal=kudu/127.9.176.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:34079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:25:53.449218 6876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:25:53.449543 6876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:25:53.449606 6876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:25:53.453820 6876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:25:53.453872 6876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:25:53.453939 6876 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.194
I20260808 22:25:53.459045 6876 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:34079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.194.keytab
--principal=kudu/127.9.176.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.176.194
--webserver_port=0
--tserver_master_addrs=127.9.176.254:43315
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6876
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.194
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:25:53.460047 6876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:25:53.461129 6876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:25:53.470696 6885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:25:53.470706 6883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:25:53.470700 6881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:25:53.470981 6876 server_base.cc:1034] running on GCE node
I20260808 22:25:53.471446 6876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:25:53.472079 6876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:25:53.473241 6876 hybrid_clock.cc:648] HybridClock initialized: now 1786227953473218 us; error 35 us; skew 500 ppm
Aug 08 22:25:53 dist-test-slave-ncp9 krb5kdc[6523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227953, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:25:53.476024 6876 init.cc:377] Logged in from keytab as kudu/127.9.176.194@KRBTEST.COM (short username kudu)
I20260808 22:25:53.476935 6876 webserver.cc:460] Webserver started at http://127.9.176.194:42669/ using document root <none> and password file <none>
I20260808 22:25:53.477483 6876 fs_manager.cc:359] Metadata directory not provided
I20260808 22:25:53.477550 6876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:25:53.477757 6876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:25:53.479764 6876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/minicluster-data/ts-1/data/instance:
uuid: "8f70d639ea46412fb961c0047840f3d9"
format_stamp: "Formatted at 2026-08-08 22:25:53 on dist-test-slave-ncp9"
server_key: "565aea2a82e8b56fbefdebecb75d2859"
server_key_iv: "ba948caeaa339a1fe6794a02bcc79fda"
server_key_version: "encryptionkey@0"
I20260808 22:25:53.480319 6876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/minicluster-data/ts-1/wal/instance:
uuid: "8f70d639ea46412fb961c0047840f3d9"
format_stamp: "Formatted at 2026-08-08 22:25:53 on dist-test-slave-ncp9"
server_key: "565aea2a82e8b56fbefdebecb75d2859"
server_key_iv: "ba948caeaa339a1fe6794a02bcc79fda"
server_key_version: "encryptionkey@0"
I20260808 22:25:53.483927 6876 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260808 22:25:53.486222 6895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:25:53.487339 6876 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 22:25:53.487447 6876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/minicluster-data/ts-1/wal
uuid: "8f70d639ea46412fb961c0047840f3d9"
format_stamp: "Formatted at 2026-08-08 22:25:53 on dist-test-slave-ncp9"
server_key: "565aea2a82e8b56fbefdebecb75d2859"
server_key_iv: "ba948caeaa339a1fe6794a02bcc79fda"
server_key_version: "encryptionkey@0"
I20260808 22:25:53.487537 6876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:25:53.509030 6876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:25:53.512205 6876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:25:53.512384 6876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:25:53.513015 6876 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:25:53.513860 6876 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:25:53.513911 6876 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:25:53.513963 6876 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:25:53.513990 6876 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:25:53.522605 6876 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.194:37235
I20260808 22:25:53.522642 7043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.194:37235 every 8 connection(s)
I20260808 22:25:53.523499 6876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
Aug 08 22:25:53 dist-test-slave-ncp9 krb5kdc[6523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227953, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.194@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
I20260808 22:25:53.533390 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 6876
I20260808 22:25:53.533529 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/minicluster-data/ts-1/wal/instance
I20260808 22:25:53.533849 9923 external_mini_cluster.cc:1428] Setting key 7c70c000a8c29f4594d7c1c69d770273
I20260808 22:25:53.536998 6643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:53.525303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.194:55771 (local address 127.9.176.254:43315)
0808 22:25:53.525431 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0808 22:25:53.525435 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:25:53.526283 (+ 848us) server_negotiation.cc:407] Connection header received
0808 22:25:53.527618 (+ 1335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:25:53.527622 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:25:53.527680 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:25:53.527758 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:25:53.528935 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:53.529467 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:53.530596 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:53.530815 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:53.530901 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:53.533430 (+ 2529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:25:53.533435 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:25:53.533438 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:25:53.533469 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:25:53.534964 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:25:53.535508 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:25:53.535511 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:25:53.535512 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:25:53.535595 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:25:53.535954 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:25:53.535956 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:25:53.535957 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:25:53.536117 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:25:53.536186 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0808 22:25:53.536749 (+ 563us) server_negotiation.cc:299] Negotiation successful
0808 22:25:53.536838 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260808 22:25:53.538013 7051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:53.525681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:43315 (local address 127.9.176.194:55771)
0808 22:25:53.526139 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:53.526182 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 22:25:53.527408 (+ 1226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:25:53.527929 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:25:53.527936 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:25:53.528291 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:25:53.528776 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:53.528790 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:53.529587 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:53.529596 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:53.530474 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:53.530483 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:53.530954 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:53.530958 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:53.531121 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:53.531681 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:25:53.531697 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:25:53.533227 (+ 1530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:25:53.535061 (+ 1834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:25:53.535067 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:25:53.535082 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:25:53.535414 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:25:53.535684 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:25:53.535687 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:25:53.535689 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:25:53.535876 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:25:53.536208 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:25:53.536215 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:25:53.536522 (+ 307us) client_negotiation.cc:769] Sending connection context
0808 22:25:53.536725 (+ 203us) client_negotiation.cc:240] Negotiation successful
0808 22:25:53.537084 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":135,"threads_started":1}
I20260808 22:25:53.539105 7047 heartbeater.cc:347] Connected to a master server at 127.9.176.254:43315
I20260808 22:25:53.539351 7047 heartbeater.cc:464] Registering TS with master...
I20260808 22:25:53.539899 7047 heartbeater.cc:511] Master 127.9.176.254:43315 requested a full tablet report, sending...
I20260808 22:25:53.541082 6588 ts_manager.cc:194] Registered new tserver with Master: 8f70d639ea46412fb961c0047840f3d9 (127.9.176.194:37235)
I20260808 22:25:53.541831 6588 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.176.194@KRBTEST.COM'} at 127.9.176.194:55771
WARNING: no policy specified for kudu/127.9.176.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.176.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.176.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.195.keytab.
Entry for principal kudu/127.9.176.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.195.keytab.
I20260808 22:25:53.559166 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.195:0
--local_ip_for_outbound_sockets=127.9.176.195
--webserver_interface=127.9.176.195
--webserver_port=0
--tserver_master_addrs=127.9.176.254:43315
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.195.keytab
--principal=kudu/127.9.176.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:34079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:25:53.668660 7055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:25:53.668911 7055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:25:53.668973 7055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:25:53.672860 7055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:25:53.672933 7055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:25:53.673005 7055 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.195
I20260808 22:25:53.678095 7055 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:34079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.195.keytab
--principal=kudu/127.9.176.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.176.195
--webserver_port=0
--tserver_master_addrs=127.9.176.254:43315
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7055
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.195
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:25:53.679076 7055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:25:53.680047 7055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:25:53.689265 7066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:25:53.689319 7055 server_base.cc:1034] running on GCE node
W20260808 22:25:53.689256 7060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:25:53.689275 7062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:25:53.689888 7055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:25:53.690486 7055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:25:53.691658 7055 hybrid_clock.cc:648] HybridClock initialized: now 1786227953691617 us; error 51 us; skew 500 ppm
Aug 08 22:25:53 dist-test-slave-ncp9 krb5kdc[6523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227953, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:25:53.694418 7055 init.cc:377] Logged in from keytab as kudu/127.9.176.195@KRBTEST.COM (short username kudu)
I20260808 22:25:53.695325 7055 webserver.cc:460] Webserver started at http://127.9.176.195:35373/ using document root <none> and password file <none>
I20260808 22:25:53.695843 7055 fs_manager.cc:359] Metadata directory not provided
I20260808 22:25:53.695904 7055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:25:53.696066 7055 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:25:53.698119 7055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/minicluster-data/ts-2/data/instance:
uuid: "33823b8b2c2844e99803d14a1f12d9fc"
format_stamp: "Formatted at 2026-08-08 22:25:53 on dist-test-slave-ncp9"
server_key: "bbe70e37fc5d2f1f4a900a8bb11f3a36"
server_key_iv: "fe2b915ecc24b45eaba4370f2d165516"
server_key_version: "encryptionkey@0"
I20260808 22:25:53.698660 7055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/minicluster-data/ts-2/wal/instance:
uuid: "33823b8b2c2844e99803d14a1f12d9fc"
format_stamp: "Formatted at 2026-08-08 22:25:53 on dist-test-slave-ncp9"
server_key: "bbe70e37fc5d2f1f4a900a8bb11f3a36"
server_key_iv: "fe2b915ecc24b45eaba4370f2d165516"
server_key_version: "encryptionkey@0"
I20260808 22:25:53.702275 7055 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 22:25:53.704988 7077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:25:53.706197 7055 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 22:25:53.706301 7055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/minicluster-data/ts-2/wal
uuid: "33823b8b2c2844e99803d14a1f12d9fc"
format_stamp: "Formatted at 2026-08-08 22:25:53 on dist-test-slave-ncp9"
server_key: "bbe70e37fc5d2f1f4a900a8bb11f3a36"
server_key_iv: "fe2b915ecc24b45eaba4370f2d165516"
server_key_version: "encryptionkey@0"
I20260808 22:25:53.706406 7055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:25:53.734562 7055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:25:53.737529 7055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:25:53.737705 7055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:25:53.738317 7055 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:25:53.739143 7055 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:25:53.739192 7055 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:25:53.739235 7055 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:25:53.739264 7055 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:25:53.747695 7055 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.195:45445
I20260808 22:25:53.747783 7242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.195:45445 every 8 connection(s)
I20260808 22:25:53.748710 7055 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
I20260808 22:25:53.754936 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 7055
I20260808 22:25:53.755070 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/minicluster-data/ts-2/wal/instance
I20260808 22:25:53.755334 9923 external_mini_cluster.cc:1428] Setting key 91cd241dd677053560ba20a19b35101c
Aug 08 22:25:53 dist-test-slave-ncp9 krb5kdc[6523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227953, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.195@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
I20260808 22:25:53.762115 6643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:53.750649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.195:56273 (local address 127.9.176.254:43315)
0808 22:25:53.750784 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0808 22:25:53.750788 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:25:53.751650 (+ 862us) server_negotiation.cc:407] Connection header received
0808 22:25:53.753009 (+ 1359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:25:53.753012 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:25:53.753068 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:25:53.753141 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:25:53.754289 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:53.754822 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:53.756237 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:53.756450 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:53.756536 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:53.758850 (+ 2314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:25:53.758853 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:25:53.758855 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:25:53.758878 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:25:53.760173 (+ 1295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:25:53.760632 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:25:53.760634 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:25:53.760636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:25:53.760708 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:25:53.761079 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:25:53.761082 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:25:53.761084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:25:53.761257 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:25:53.761343 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0808 22:25:53.761820 (+ 477us) server_negotiation.cc:299] Negotiation successful
0808 22:25:53.761935 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 22:25:53.762859 7245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:53.751068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:43315 (local address 127.9.176.195:56273)
0808 22:25:53.751508 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:53.751551 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 22:25:53.752780 (+ 1229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:25:53.753316 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:25:53.753323 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:25:53.753666 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:25:53.754147 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:53.754161 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:53.754938 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:53.754941 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:53.756043 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:53.756055 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:53.756589 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:53.756593 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:53.756766 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:53.757354 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:25:53.757370 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:25:53.758703 (+ 1333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:25:53.760274 (+ 1571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:25:53.760280 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:25:53.760294 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:25:53.760533 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:25:53.760790 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:25:53.760793 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:25:53.760795 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:25:53.760974 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:25:53.761354 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:25:53.761361 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:25:53.761587 (+ 226us) client_negotiation.cc:769] Sending connection context
0808 22:25:53.761743 (+ 156us) client_negotiation.cc:240] Negotiation successful
0808 22:25:53.762043 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":121,"threads_started":1}
I20260808 22:25:53.763818 7243 heartbeater.cc:347] Connected to a master server at 127.9.176.254:43315
I20260808 22:25:53.764031 7243 heartbeater.cc:464] Registering TS with master...
I20260808 22:25:53.764487 7243 heartbeater.cc:511] Master 127.9.176.254:43315 requested a full tablet report, sending...
I20260808 22:25:53.765422 6588 ts_manager.cc:194] Registered new tserver with Master: 33823b8b2c2844e99803d14a1f12d9fc (127.9.176.195:45445)
I20260808 22:25:53.765949 6588 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.176.195@KRBTEST.COM'} at 127.9.176.195:56273
I20260808 22:25:53.768738 9923 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 22:25:53.777062 6643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:53.770658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49076 (local address 127.9.176.254:43315)
0808 22:25:53.770768 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0808 22:25:53.770772 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:25:53.770949 (+ 177us) server_negotiation.cc:407] Connection header received
0808 22:25:53.771041 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:25:53.771043 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:25:53.771079 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:25:53.771142 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:25:53.771732 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:53.772209 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:53.772997 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:53.773193 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:53.773263 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:53.774412 (+ 1149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:25:53.774416 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:25:53.774418 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:25:53.774443 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:25:53.775739 (+ 1296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:25:53.776066 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:25:53.776068 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:25:53.776069 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:25:53.776133 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:25:53.776354 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:25:53.776356 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:25:53.776357 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:25:53.776575 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:25:53.776673 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0808 22:25:53.776838 (+ 165us) server_negotiation.cc:299] Negotiation successful
0808 22:25:53.776953 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 22:25:53.779501 6588 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49076:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 22:25:53.782554 6588 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 22:25:53.796654 7255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:53.790247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.193:46441 (local address 127.0.0.1:39152)
0808 22:25:53.791009 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:53.791059 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0808 22:25:53.791186 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:25:53.792130 (+ 944us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:25:53.792136 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:25:53.792176 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:25:53.792307 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:53.792314 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:53.794121 (+ 1807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:53.794125 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:53.795125 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:53.795135 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:53.796108 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:53.796112 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:53.796226 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:53.796267 (+ 41us) client_negotiation.cc:769] Sending connection context
0808 22:25:53.796377 (+ 110us) client_negotiation.cc:240] Negotiation successful
0808 22:25:53.796498 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"mutex_wait_us":4,"spinlock_wait_cycles":11776,"thread_start_us":79,"threads_started":1}
I20260808 22:25:53.796696 7257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:53.790454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39152 (local address 127.9.176.193:46441)
0808 22:25:53.791669 (+ 1215us) server_negotiation.cc:206] Beginning negotiation
0808 22:25:53.791685 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 22:25:53.791708 (+ 23us) server_negotiation.cc:407] Connection header received
0808 22:25:53.791792 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:25:53.791798 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:25:53.791985 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:25:53.792135 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:25:53.792433 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:53.793678 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:53.795244 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:53.795985 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:53.796092 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:53.796238 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0808 22:25:53.796425 (+ 187us) server_negotiation.cc:299] Negotiation successful
0808 22:25:53.796527 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1120,"thread_start_us":81,"threads_started":1}
I20260808 22:25:53.796955 7260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:53.791097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50028 (local address 127.9.176.195:45445)
0808 22:25:53.791689 (+ 592us) server_negotiation.cc:206] Beginning negotiation
0808 22:25:53.791701 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 22:25:53.791724 (+ 23us) server_negotiation.cc:407] Connection header received
0808 22:25:53.791804 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:25:53.791809 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:25:53.791986 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:25:53.792136 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:25:53.792477 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:53.793660 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:53.795551 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:53.796120 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:53.796224 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:53.796360 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0808 22:25:53.796676 (+ 316us) server_negotiation.cc:299] Negotiation successful
0808 22:25:53.796775 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":133,"threads_started":1}
I20260808 22:25:53.797350 7256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:53.790646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.195:45445 (local address 127.0.0.1:50028)
0808 22:25:53.791385 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:53.791395 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 22:25:53.791471 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:25:53.792130 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:25:53.792135 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:25:53.792171 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:25:53.792295 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:53.792303 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:53.793806 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:53.793809 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:53.795399 (+ 1590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:53.795410 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:53.796420 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:53.796424 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:53.796511 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:53.796532 (+ 21us) client_negotiation.cc:769] Sending connection context
0808 22:25:53.797147 (+ 615us) client_negotiation.cc:240] Negotiation successful
0808 22:25:53.797225 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":693,"mutex_wait_us":116,"spinlock_wait_cycles":3968,"thread_start_us":90,"threads_started":1}
I20260808 22:25:53.797647 7259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:53.791045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53526 (local address 127.9.176.194:37235)
0808 22:25:53.792433 (+ 1388us) server_negotiation.cc:206] Beginning negotiation
0808 22:25:53.792445 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 22:25:53.792465 (+ 20us) server_negotiation.cc:407] Connection header received
0808 22:25:53.792534 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:25:53.792539 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:25:53.792671 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:25:53.792791 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:25:53.793029 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:53.793780 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:53.795523 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:53.796317 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:53.796410 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:53.796532 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0808 22:25:53.797413 (+ 881us) server_negotiation.cc:299] Negotiation successful
0808 22:25:53.797503 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1289,"thread_start_us":131,"threads_started":1}
I20260808 22:25:53.798271 7258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:53.790390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.194:37235 (local address 127.0.0.1:53526)
0808 22:25:53.791144 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:53.791155 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 22:25:53.791222 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:25:53.792786 (+ 1564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:25:53.792788 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:25:53.792804 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:25:53.792933 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:53.792940 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:53.793916 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:53.793920 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:53.795398 (+ 1478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:53.795410 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:53.796597 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:53.796600 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:53.796697 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:53.797161 (+ 464us) client_negotiation.cc:769] Sending connection context
0808 22:25:53.797238 (+ 77us) client_negotiation.cc:240] Negotiation successful
0808 22:25:53.797310 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":682,"mutex_wait_us":121,"thread_start_us":79,"threads_started":1}
I20260808 22:25:53.799840 6773 tablet_service.cc:1467] Processing CreateTablet for tablet db620cc9372649c79a86e438c2e02271 (DEFAULT_TABLE table=test-table [id=fd34442726f5468f931f3e8ea92b11ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:25:53.800015 7151 tablet_service.cc:1467] Processing CreateTablet for tablet db620cc9372649c79a86e438c2e02271 (DEFAULT_TABLE table=test-table [id=fd34442726f5468f931f3e8ea92b11ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:25:53.800751 6773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db620cc9372649c79a86e438c2e02271. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:25:53.801075 7151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db620cc9372649c79a86e438c2e02271. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:25:53.800808 6959 tablet_service.cc:1467] Processing CreateTablet for tablet db620cc9372649c79a86e438c2e02271 (DEFAULT_TABLE table=test-table [id=fd34442726f5468f931f3e8ea92b11ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:25:53.801800 6959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db620cc9372649c79a86e438c2e02271. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:25:53.806999 7261 tablet_bootstrap.cc:492] T db620cc9372649c79a86e438c2e02271 P d448ff9563af42e3bf7aaac8f9856c0d: Bootstrap starting.
I20260808 22:25:53.808872 7264 tablet_bootstrap.cc:492] T db620cc9372649c79a86e438c2e02271 P 8f70d639ea46412fb961c0047840f3d9: Bootstrap starting.
I20260808 22:25:53.809242 7263 tablet_bootstrap.cc:492] T db620cc9372649c79a86e438c2e02271 P 33823b8b2c2844e99803d14a1f12d9fc: Bootstrap starting.
I20260808 22:25:53.809480 7261 tablet_bootstrap.cc:654] T db620cc9372649c79a86e438c2e02271 P d448ff9563af42e3bf7aaac8f9856c0d: Neither blocks nor log segments found. Creating new log.
I20260808 22:25:53.810854 7261 log.cc:824] T db620cc9372649c79a86e438c2e02271 P d448ff9563af42e3bf7aaac8f9856c0d: Log is configured to *not* fsync() on all Append() calls
I20260808 22:25:53.811187 7264 tablet_bootstrap.cc:654] T db620cc9372649c79a86e438c2e02271 P 8f70d639ea46412fb961c0047840f3d9: Neither blocks nor log segments found. Creating new log.
I20260808 22:25:53.811342 7263 tablet_bootstrap.cc:654] T db620cc9372649c79a86e438c2e02271 P 33823b8b2c2844e99803d14a1f12d9fc: Neither blocks nor log segments found. Creating new log.
I20260808 22:25:53.811919 7264 log.cc:824] T db620cc9372649c79a86e438c2e02271 P 8f70d639ea46412fb961c0047840f3d9: Log is configured to *not* fsync() on all Append() calls
I20260808 22:25:53.812515 7263 log.cc:824] T db620cc9372649c79a86e438c2e02271 P 33823b8b2c2844e99803d14a1f12d9fc: Log is configured to *not* fsync() on all Append() calls
I20260808 22:25:53.813103 7261 tablet_bootstrap.cc:492] T db620cc9372649c79a86e438c2e02271 P d448ff9563af42e3bf7aaac8f9856c0d: No bootstrap required, opened a new log
I20260808 22:25:53.813302 7261 ts_tablet_manager.cc:1397] T db620cc9372649c79a86e438c2e02271 P d448ff9563af42e3bf7aaac8f9856c0d: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260808 22:25:53.814278 7263 tablet_bootstrap.cc:492] T db620cc9372649c79a86e438c2e02271 P 33823b8b2c2844e99803d14a1f12d9fc: No bootstrap required, opened a new log
I20260808 22:25:53.814451 7263 ts_tablet_manager.cc:1397] T db620cc9372649c79a86e438c2e02271 P 33823b8b2c2844e99803d14a1f12d9fc: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260808 22:25:53.814461 7264 tablet_bootstrap.cc:492] T db620cc9372649c79a86e438c2e02271 P 8f70d639ea46412fb961c0047840f3d9: No bootstrap required, opened a new log
I20260808 22:25:53.814620 7264 ts_tablet_manager.cc:1397] T db620cc9372649c79a86e438c2e02271 P 8f70d639ea46412fb961c0047840f3d9: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260808 22:25:53.817142 7263 raft_consensus.cc:348] T db620cc9372649c79a86e438c2e02271 P 33823b8b2c2844e99803d14a1f12d9fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d448ff9563af42e3bf7aaac8f9856c0d" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 46441 } } peers { permanent_uuid: "8f70d639ea46412fb961c0047840f3d9" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 37235 } } peers { permanent_uuid: "33823b8b2c2844e99803d14a1f12d9fc" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45445 } }
I20260808 22:25:53.817137 7261 raft_consensus.cc:348] T db620cc9372649c79a86e438c2e02271 P d448ff9563af42e3bf7aaac8f9856c0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d448ff9563af42e3bf7aaac8f9856c0d" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 46441 } } peers { permanent_uuid: "8f70d639ea46412fb961c0047840f3d9" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 37235 } } peers { permanent_uuid: "33823b8b2c2844e99803d14a1f12d9fc" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45445 } }
I20260808 22:25:53.817377 7261 raft_consensus.cc:374] T db620cc9372649c79a86e438c2e02271 P d448ff9563af42e3bf7aaac8f9856c0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:25:53.817432 7261 raft_consensus.cc:729] T db620cc9372649c79a86e438c2e02271 P d448ff9563af42e3bf7aaac8f9856c0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d448ff9563af42e3bf7aaac8f9856c0d, State: Initialized, Role: FOLLOWER
I20260808 22:25:53.817548 7263 raft_consensus.cc:374] T db620cc9372649c79a86e438c2e02271 P 33823b8b2c2844e99803d14a1f12d9fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:25:53.817611 7263 raft_consensus.cc:729] T db620cc9372649c79a86e438c2e02271 P 33823b8b2c2844e99803d14a1f12d9fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33823b8b2c2844e99803d14a1f12d9fc, State: Initialized, Role: FOLLOWER
I20260808 22:25:53.817937 7261 consensus_queue.cc:260] T db620cc9372649c79a86e438c2e02271 P d448ff9563af42e3bf7aaac8f9856c0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d448ff9563af42e3bf7aaac8f9856c0d" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 46441 } } peers { permanent_uuid: "8f70d639ea46412fb961c0047840f3d9" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 37235 } } peers { permanent_uuid: "33823b8b2c2844e99803d14a1f12d9fc" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45445 } }
I20260808 22:25:53.818041 7263 consensus_queue.cc:260] T db620cc9372649c79a86e438c2e02271 P 33823b8b2c2844e99803d14a1f12d9fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d448ff9563af42e3bf7aaac8f9856c0d" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 46441 } } peers { permanent_uuid: "8f70d639ea46412fb961c0047840f3d9" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 37235 } } peers { permanent_uuid: "33823b8b2c2844e99803d14a1f12d9fc" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45445 } }
I20260808 22:25:53.818188 7264 raft_consensus.cc:348] T db620cc9372649c79a86e438c2e02271 P 8f70d639ea46412fb961c0047840f3d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d448ff9563af42e3bf7aaac8f9856c0d" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 46441 } } peers { permanent_uuid: "8f70d639ea46412fb961c0047840f3d9" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 37235 } } peers { permanent_uuid: "33823b8b2c2844e99803d14a1f12d9fc" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45445 } }
I20260808 22:25:53.818418 7264 raft_consensus.cc:374] T db620cc9372649c79a86e438c2e02271 P 8f70d639ea46412fb961c0047840f3d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:25:53.818459 7264 raft_consensus.cc:729] T db620cc9372649c79a86e438c2e02271 P 8f70d639ea46412fb961c0047840f3d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f70d639ea46412fb961c0047840f3d9, State: Initialized, Role: FOLLOWER
I20260808 22:25:53.818831 7264 consensus_queue.cc:260] T db620cc9372649c79a86e438c2e02271 P 8f70d639ea46412fb961c0047840f3d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d448ff9563af42e3bf7aaac8f9856c0d" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 46441 } } peers { permanent_uuid: "8f70d639ea46412fb961c0047840f3d9" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 37235 } } peers { permanent_uuid: "33823b8b2c2844e99803d14a1f12d9fc" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45445 } }
I20260808 22:25:53.818949 7261 ts_tablet_manager.cc:1428] T db620cc9372649c79a86e438c2e02271 P d448ff9563af42e3bf7aaac8f9856c0d: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260808 22:25:53.818953 7263 ts_tablet_manager.cc:1428] T db620cc9372649c79a86e438c2e02271 P 33823b8b2c2844e99803d14a1f12d9fc: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260808 22:25:53.819154 7243 heartbeater.cc:503] Master 127.9.176.254:43315 was elected leader, sending a full tablet report...
I20260808 22:25:53.819263 6869 heartbeater.cc:503] Master 127.9.176.254:43315 was elected leader, sending a full tablet report...
I20260808 22:25:53.819657 7264 ts_tablet_manager.cc:1428] T db620cc9372649c79a86e438c2e02271 P 8f70d639ea46412fb961c0047840f3d9: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260808 22:25:53.819983 7047 heartbeater.cc:503] Master 127.9.176.254:43315 was elected leader, sending a full tablet report...
W20260808 22:25:53.999846 7244 tablet.cc:2369] T db620cc9372649c79a86e438c2e02271 P 33823b8b2c2844e99803d14a1f12d9fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 22:25:54.024633 7049 tablet.cc:2369] T db620cc9372649c79a86e438c2e02271 P 8f70d639ea46412fb961c0047840f3d9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 22:25:54.048491 7272 raft_consensus.cc:482] T db620cc9372649c79a86e438c2e02271 P 8f70d639ea46412fb961c0047840f3d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 22:25:54.048681 7272 raft_consensus.cc:504] T db620cc9372649c79a86e438c2e02271 P 8f70d639ea46412fb961c0047840f3d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d448ff9563af42e3bf7aaac8f9856c0d" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 46441 } } peers { permanent_uuid: "8f70d639ea46412fb961c0047840f3d9" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 37235 } } peers { permanent_uuid: "33823b8b2c2844e99803d14a1f12d9fc" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45445 } }
I20260808 22:25:54.049784 7272 leader_election.cc:288] T db620cc9372649c79a86e438c2e02271 P 8f70d639ea46412fb961c0047840f3d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d448ff9563af42e3bf7aaac8f9856c0d (127.9.176.193:46441), 33823b8b2c2844e99803d14a1f12d9fc (127.9.176.195:45445)
I20260808 22:25:54.050542 7270 raft_consensus.cc:482] T db620cc9372649c79a86e438c2e02271 P d448ff9563af42e3bf7aaac8f9856c0d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 22:25:54.050746 7270 raft_consensus.cc:504] T db620cc9372649c79a86e438c2e02271 P d448ff9563af42e3bf7aaac8f9856c0d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d448ff9563af42e3bf7aaac8f9856c0d" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 46441 } } peers { permanent_uuid: "8f70d639ea46412fb961c0047840f3d9" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 37235 } } peers { permanent_uuid: "33823b8b2c2844e99803d14a1f12d9fc" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45445 } }
I20260808 22:25:54.051803 7270 leader_election.cc:288] T db620cc9372649c79a86e438c2e02271 P d448ff9563af42e3bf7aaac8f9856c0d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f70d639ea46412fb961c0047840f3d9 (127.9.176.194:37235), 33823b8b2c2844e99803d14a1f12d9fc (127.9.176.195:45445)
W20260808 22:25:54.054554 6872 tablet.cc:2369] T db620cc9372649c79a86e438c2e02271 P d448ff9563af42e3bf7aaac8f9856c0d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 22:25:54.055168 7260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:54.050152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.194:50745 (local address 127.9.176.195:45445)
0808 22:25:54.050316 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0808 22:25:54.050320 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:25:54.050532 (+ 212us) server_negotiation.cc:407] Connection header received
0808 22:25:54.050763 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:25:54.050767 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:25:54.050818 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:25:54.050863 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:25:54.051283 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:54.051971 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:54.053880 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:54.054506 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:54.054577 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:54.054622 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0808 22:25:54.054956 (+ 334us) server_negotiation.cc:299] Negotiation successful
0808 22:25:54.055034 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260808 22:25:54.055205 7257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:54.050097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.194:59281 (local address 127.9.176.193:46441)
0808 22:25:54.050225 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0808 22:25:54.050229 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:25:54.050369 (+ 140us) server_negotiation.cc:407] Connection header received
0808 22:25:54.050573 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:25:54.050576 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:25:54.050625 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:25:54.050694 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:25:54.051051 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:54.051769 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:54.053895 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:54.054478 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:54.054568 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:54.054623 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0808 22:25:54.054962 (+ 339us) server_negotiation.cc:299] Negotiation successful
0808 22:25:54.055106 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":48}
I20260808 22:25:54.055569 7274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:54.050039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.195:45445 (local address 127.9.176.194:50745)
0808 22:25:54.050432 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:54.050445 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 22:25:54.050613 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:25:54.051008 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:25:54.051011 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:25:54.051029 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:25:54.051152 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:54.051159 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:54.052111 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:54.052115 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:54.053754 (+ 1639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:54.053762 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:54.054717 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:54.054721 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:54.054818 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:54.054841 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 22:25:54.055347 (+ 506us) client_negotiation.cc:240] Negotiation successful
0808 22:25:54.055424 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"mutex_wait_us":74,"thread_start_us":114,"threads_started":1}
I20260808 22:25:54.055860 7273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:54.049991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.193:46441 (local address 127.9.176.194:59281)
0808 22:25:54.050288 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:54.050303 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 22:25:54.050452 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:25:54.050791 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:25:54.050795 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:25:54.050814 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:25:54.050936 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:54.050942 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:54.052296 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:54.052300 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:54.053785 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:54.053790 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:54.054750 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:54.054753 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:54.054845 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:54.054862 (+ 17us) client_negotiation.cc:769] Sending connection context
0808 22:25:54.054997 (+ 135us) client_negotiation.cc:240] Negotiation successful
0808 22:25:54.055577 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"mutex_wait_us":125,"thread_start_us":69,"threads_started":1}
I20260808 22:25:54.056218 7178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db620cc9372649c79a86e438c2e02271" candidate_uuid: "8f70d639ea46412fb961c0047840f3d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33823b8b2c2844e99803d14a1f12d9fc" is_pre_election: true
I20260808 22:25:54.056524 7178 raft_consensus.cc:2454] T db620cc9372649c79a86e438c2e02271 P 33823b8b2c2844e99803d14a1f12d9fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f70d639ea46412fb961c0047840f3d9 in term 0.
I20260808 22:25:54.057123 7259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:54.052045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.193:37591 (local address 127.9.176.194:37235)
0808 22:25:54.052342 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0808 22:25:54.052346 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:25:54.052362 (+ 16us) server_negotiation.cc:407] Connection header received
0808 22:25:54.052511 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:25:54.052514 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:25:54.052554 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:25:54.052596 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:25:54.052977 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:54.053668 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:54.055973 (+ 2305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:54.056481 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:54.056592 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:54.056634 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0808 22:25:54.056939 (+ 305us) server_negotiation.cc:299] Negotiation successful
0808 22:25:54.057011 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232}
I20260808 22:25:54.057132 7275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:54.051944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.194:37235 (local address 127.9.176.193:37591)
0808 22:25:54.052296 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:54.052309 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 22:25:54.052399 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:25:54.052715 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:25:54.052718 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:25:54.052738 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:25:54.052848 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:54.052853 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:54.053773 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:54.053776 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:54.055841 (+ 2065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:54.055852 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:54.056697 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:54.056701 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:54.056792 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:54.056816 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 22:25:54.056919 (+ 103us) client_negotiation.cc:240] Negotiation successful
0808 22:25:54.056989 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":117,"threads_started":1}
I20260808 22:25:54.057406 6900 leader_election.cc:302] T db620cc9372649c79a86e438c2e02271 P 8f70d639ea46412fb961c0047840f3d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33823b8b2c2844e99803d14a1f12d9fc, 8f70d639ea46412fb961c0047840f3d9; no voters:
I20260808 22:25:54.057615 7277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:54.052245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.193:41221 (local address 127.9.176.195:45445)
0808 22:25:54.052583 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0808 22:25:54.052604 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0808 22:25:54.053452 (+ 848us) server_negotiation.cc:407] Connection header received
0808 22:25:54.053497 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:25:54.053500 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:25:54.053538 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:25:54.053602 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:25:54.053964 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:54.054734 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:54.056105 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:54.056646 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:54.056747 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:54.056792 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0808 22:25:54.057422 (+ 630us) server_negotiation.cc:299] Negotiation successful
0808 22:25:54.057489 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":275,"thread_start_us":106,"threads_started":1}
I20260808 22:25:54.057579 6986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db620cc9372649c79a86e438c2e02271" candidate_uuid: "d448ff9563af42e3bf7aaac8f9856c0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f70d639ea46412fb961c0047840f3d9" is_pre_election: true
I20260808 22:25:54.057686 7272 raft_consensus.cc:2781] T db620cc9372649c79a86e438c2e02271 P 8f70d639ea46412fb961c0047840f3d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 22:25:54.057137 6802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db620cc9372649c79a86e438c2e02271" candidate_uuid: "8f70d639ea46412fb961c0047840f3d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d448ff9563af42e3bf7aaac8f9856c0d" is_pre_election: true
I20260808 22:25:54.057773 7272 raft_consensus.cc:482] T db620cc9372649c79a86e438c2e02271 P 8f70d639ea46412fb961c0047840f3d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 22:25:54.057814 7272 raft_consensus.cc:3037] T db620cc9372649c79a86e438c2e02271 P 8f70d639ea46412fb961c0047840f3d9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:25:54.057854 7276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:54.052037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.195:45445 (local address 127.9.176.193:41221)
0808 22:25:54.053261 (+ 1224us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:54.053274 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 22:25:54.053378 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:25:54.053705 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:25:54.053708 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:25:54.053726 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:25:54.053841 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:54.053847 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:54.054856 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:54.054860 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:54.056005 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:54.056012 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:54.056817 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:54.056821 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:54.056943 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:54.056969 (+ 26us) client_negotiation.cc:769] Sending connection context
0808 22:25:54.057037 (+ 68us) client_negotiation.cc:240] Negotiation successful
0808 22:25:54.057102 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1164,"mutex_wait_us":3,"thread_start_us":107,"threads_started":1}
I20260808 22:25:54.057847 6802 raft_consensus.cc:2454] T db620cc9372649c79a86e438c2e02271 P d448ff9563af42e3bf7aaac8f9856c0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f70d639ea46412fb961c0047840f3d9 in term 0.
I20260808 22:25:54.058530 7178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db620cc9372649c79a86e438c2e02271" candidate_uuid: "d448ff9563af42e3bf7aaac8f9856c0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33823b8b2c2844e99803d14a1f12d9fc" is_pre_election: true
I20260808 22:25:54.058636 7178 raft_consensus.cc:2454] T db620cc9372649c79a86e438c2e02271 P 33823b8b2c2844e99803d14a1f12d9fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d448ff9563af42e3bf7aaac8f9856c0d in term 0.
I20260808 22:25:54.058959 6714 leader_election.cc:302] T db620cc9372649c79a86e438c2e02271 P d448ff9563af42e3bf7aaac8f9856c0d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33823b8b2c2844e99803d14a1f12d9fc, d448ff9563af42e3bf7aaac8f9856c0d; no voters:
I20260808 22:25:54.059010 7272 raft_consensus.cc:504] T db620cc9372649c79a86e438c2e02271 P 8f70d639ea46412fb961c0047840f3d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d448ff9563af42e3bf7aaac8f9856c0d" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 46441 } } peers { permanent_uuid: "8f70d639ea46412fb961c0047840f3d9" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 37235 } } peers { permanent_uuid: "33823b8b2c2844e99803d14a1f12d9fc" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45445 } }
I20260808 22:25:54.059176 7270 raft_consensus.cc:2781] T db620cc9372649c79a86e438c2e02271 P d448ff9563af42e3bf7aaac8f9856c0d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 22:25:54.059268 7270 raft_consensus.cc:482] T db620cc9372649c79a86e438c2e02271 P d448ff9563af42e3bf7aaac8f9856c0d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 22:25:54.059260 6986 raft_consensus.cc:2379] T db620cc9372649c79a86e438c2e02271 P 8f70d639ea46412fb961c0047840f3d9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d448ff9563af42e3bf7aaac8f9856c0d in current term 1: Already voted for candidate 8f70d639ea46412fb961c0047840f3d9 in this term.
I20260808 22:25:54.059321 7270 raft_consensus.cc:3037] T db620cc9372649c79a86e438c2e02271 P d448ff9563af42e3bf7aaac8f9856c0d [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:25:54.059521 7272 leader_election.cc:288] T db620cc9372649c79a86e438c2e02271 P 8f70d639ea46412fb961c0047840f3d9 [CANDIDATE]: Term 1 election: Requested vote from peers d448ff9563af42e3bf7aaac8f9856c0d (127.9.176.193:46441), 33823b8b2c2844e99803d14a1f12d9fc (127.9.176.195:45445)
I20260808 22:25:54.060316 6802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db620cc9372649c79a86e438c2e02271" candidate_uuid: "8f70d639ea46412fb961c0047840f3d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d448ff9563af42e3bf7aaac8f9856c0d"
I20260808 22:25:54.060336 7178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db620cc9372649c79a86e438c2e02271" candidate_uuid: "8f70d639ea46412fb961c0047840f3d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33823b8b2c2844e99803d14a1f12d9fc"
I20260808 22:25:54.060439 7178 raft_consensus.cc:3037] T db620cc9372649c79a86e438c2e02271 P 33823b8b2c2844e99803d14a1f12d9fc [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:25:54.060487 7270 raft_consensus.cc:504] T db620cc9372649c79a86e438c2e02271 P d448ff9563af42e3bf7aaac8f9856c0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d448ff9563af42e3bf7aaac8f9856c0d" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 46441 } } peers { permanent_uuid: "8f70d639ea46412fb961c0047840f3d9" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 37235 } } peers { permanent_uuid: "33823b8b2c2844e99803d14a1f12d9fc" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45445 } }
I20260808 22:25:54.060706 6802 raft_consensus.cc:2379] T db620cc9372649c79a86e438c2e02271 P d448ff9563af42e3bf7aaac8f9856c0d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8f70d639ea46412fb961c0047840f3d9 in current term 1: Already voted for candidate d448ff9563af42e3bf7aaac8f9856c0d in this term.
I20260808 22:25:54.060856 7270 leader_election.cc:288] T db620cc9372649c79a86e438c2e02271 P d448ff9563af42e3bf7aaac8f9856c0d [CANDIDATE]: Term 1 election: Requested vote from peers 8f70d639ea46412fb961c0047840f3d9 (127.9.176.194:37235), 33823b8b2c2844e99803d14a1f12d9fc (127.9.176.195:45445)
I20260808 22:25:54.061332 7176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db620cc9372649c79a86e438c2e02271" candidate_uuid: "d448ff9563af42e3bf7aaac8f9856c0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33823b8b2c2844e99803d14a1f12d9fc"
I20260808 22:25:54.061689 7178 raft_consensus.cc:2454] T db620cc9372649c79a86e438c2e02271 P 33823b8b2c2844e99803d14a1f12d9fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f70d639ea46412fb961c0047840f3d9 in term 1.
I20260808 22:25:54.061913 6900 leader_election.cc:302] T db620cc9372649c79a86e438c2e02271 P 8f70d639ea46412fb961c0047840f3d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 33823b8b2c2844e99803d14a1f12d9fc, 8f70d639ea46412fb961c0047840f3d9; no voters: d448ff9563af42e3bf7aaac8f9856c0d
I20260808 22:25:54.062034 6986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db620cc9372649c79a86e438c2e02271" candidate_uuid: "d448ff9563af42e3bf7aaac8f9856c0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f70d639ea46412fb961c0047840f3d9"
I20260808 22:25:54.062084 7272 raft_consensus.cc:2781] T db620cc9372649c79a86e438c2e02271 P 8f70d639ea46412fb961c0047840f3d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:25:54.062278 7272 raft_consensus.cc:686] T db620cc9372649c79a86e438c2e02271 P 8f70d639ea46412fb961c0047840f3d9 [term 1 LEADER]: Becoming Leader. State: Replica: 8f70d639ea46412fb961c0047840f3d9, State: Running, Role: LEADER
I20260808 22:25:54.062562 7272 consensus_queue.cc:237] T db620cc9372649c79a86e438c2e02271 P 8f70d639ea46412fb961c0047840f3d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d448ff9563af42e3bf7aaac8f9856c0d" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 46441 } } peers { permanent_uuid: "8f70d639ea46412fb961c0047840f3d9" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 37235 } } peers { permanent_uuid: "33823b8b2c2844e99803d14a1f12d9fc" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45445 } }
I20260808 22:25:54.064273 6713 leader_election.cc:302] T db620cc9372649c79a86e438c2e02271 P d448ff9563af42e3bf7aaac8f9856c0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d448ff9563af42e3bf7aaac8f9856c0d; no voters: 33823b8b2c2844e99803d14a1f12d9fc, 8f70d639ea46412fb961c0047840f3d9
I20260808 22:25:54.064476 7270 raft_consensus.cc:2726] T db620cc9372649c79a86e438c2e02271 P d448ff9563af42e3bf7aaac8f9856c0d [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260808 22:25:54.065507 6586 catalog_manager.cc:5495] T db620cc9372649c79a86e438c2e02271 P 8f70d639ea46412fb961c0047840f3d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f70d639ea46412fb961c0047840f3d9 (127.9.176.194). New cstate: current_term: 1 leader_uuid: "8f70d639ea46412fb961c0047840f3d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d448ff9563af42e3bf7aaac8f9856c0d" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 46441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f70d639ea46412fb961c0047840f3d9" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 37235 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "33823b8b2c2844e99803d14a1f12d9fc" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45445 } health_report { overall_health: UNKNOWN } } }
I20260808 22:25:54.089627 7259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:54.086650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53534 (local address 127.9.176.194:37235)
0808 22:25:54.086771 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0808 22:25:54.086774 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:25:54.086787 (+ 13us) server_negotiation.cc:407] Connection header received
0808 22:25:54.086907 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:25:54.086909 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:25:54.086958 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:25:54.087041 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:25:54.087212 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:54.087686 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:54.088433 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:54.088627 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:54.088688 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:54.088932 (+ 244us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:25:54.089324 (+ 392us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:25:54.089412 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0808 22:25:54.089459 (+ 47us) server_negotiation.cc:299] Negotiation successful
0808 22:25:54.089501 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 22:25:54.095031 7178 raft_consensus.cc:1261] T db620cc9372649c79a86e438c2e02271 P 33823b8b2c2844e99803d14a1f12d9fc [term 1 FOLLOWER]: Refusing update from remote peer 8f70d639ea46412fb961c0047840f3d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:25:54.095034 6802 raft_consensus.cc:1261] T db620cc9372649c79a86e438c2e02271 P d448ff9563af42e3bf7aaac8f9856c0d [term 1 FOLLOWER]: Refusing update from remote peer 8f70d639ea46412fb961c0047840f3d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:25:54.095587 7272 consensus_queue.cc:1035] T db620cc9372649c79a86e438c2e02271 P 8f70d639ea46412fb961c0047840f3d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d448ff9563af42e3bf7aaac8f9856c0d" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 46441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:25:54.095799 7280 consensus_queue.cc:1035] T db620cc9372649c79a86e438c2e02271 P 8f70d639ea46412fb961c0047840f3d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33823b8b2c2844e99803d14a1f12d9fc" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:25:54.105587 7287 mvcc.cc:205] Tried to move back new op lower bound from 7316389699966799872 to 7316389699842666496. Current Snapshot: MvccSnapshot[applied={T|T < 7316389699966799872}]
I20260808 22:25:54.109505 6587 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49076:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:25:54.109649 6587 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49076:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:25:54.111768 6587 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fd986986e209479fb3bbbf41c958f233: Sending DeleteTablet for 3 replicas of tablet db620cc9372649c79a86e438c2e02271
I20260808 22:25:54.112452 6773 tablet_service.cc:1514] Processing DeleteTablet for tablet db620cc9372649c79a86e438c2e02271 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:25:54 UTC) from {username='kudu', principal='kudu/127.9.176.254@KRBTEST.COM'} at 127.0.0.1:39152
I20260808 22:25:54.112504 7151 tablet_service.cc:1514] Processing DeleteTablet for tablet db620cc9372649c79a86e438c2e02271 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:25:54 UTC) from {username='kudu', principal='kudu/127.9.176.254@KRBTEST.COM'} at 127.0.0.1:50028
I20260808 22:25:54.112975 7299 tablet_replica.cc:330] T db620cc9372649c79a86e438c2e02271 P 33823b8b2c2844e99803d14a1f12d9fc: stopping tablet replica
I20260808 22:25:54.112951 6959 tablet_service.cc:1514] Processing DeleteTablet for tablet db620cc9372649c79a86e438c2e02271 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:25:54 UTC) from {username='kudu', principal='kudu/127.9.176.254@KRBTEST.COM'} at 127.0.0.1:53526
I20260808 22:25:54.113277 7299 raft_consensus.cc:2229] T db620cc9372649c79a86e438c2e02271 P 33823b8b2c2844e99803d14a1f12d9fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:25:54.113389 7305 tablet_replica.cc:330] T db620cc9372649c79a86e438c2e02271 P 8f70d639ea46412fb961c0047840f3d9: stopping tablet replica
I20260808 22:25:54.113574 7299 raft_consensus.cc:2258] T db620cc9372649c79a86e438c2e02271 P 33823b8b2c2844e99803d14a1f12d9fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:25:54.113750 7305 raft_consensus.cc:2229] T db620cc9372649c79a86e438c2e02271 P 8f70d639ea46412fb961c0047840f3d9 [term 1 LEADER]: Raft consensus shutting down.
I20260808 22:25:54.114207 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 6687
I20260808 22:25:54.117340 7305 raft_consensus.cc:2258] T db620cc9372649c79a86e438c2e02271 P 8f70d639ea46412fb961c0047840f3d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:25:54.117522 7299 ts_tablet_manager.cc:1905] T db620cc9372649c79a86e438c2e02271 P 33823b8b2c2844e99803d14a1f12d9fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:25:54.120306 7305 ts_tablet_manager.cc:1905] T db620cc9372649c79a86e438c2e02271 P 8f70d639ea46412fb961c0047840f3d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:25:54.121187 7299 ts_tablet_manager.cc:1918] T db620cc9372649c79a86e438c2e02271 P 33823b8b2c2844e99803d14a1f12d9fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 22:25:54.121286 7299 log.cc:1196] T db620cc9372649c79a86e438c2e02271 P 33823b8b2c2844e99803d14a1f12d9fc: Deleting WAL directory at /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/minicluster-data/ts-2/wal/wals/db620cc9372649c79a86e438c2e02271
I20260808 22:25:54.121603 7299 ts_tablet_manager.cc:1939] T db620cc9372649c79a86e438c2e02271 P 33823b8b2c2844e99803d14a1f12d9fc: Deleting consensus metadata
I20260808 22:25:54.123122 7305 ts_tablet_manager.cc:1918] T db620cc9372649c79a86e438c2e02271 P 8f70d639ea46412fb961c0047840f3d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 22:25:54.123214 7305 log.cc:1196] T db620cc9372649c79a86e438c2e02271 P 8f70d639ea46412fb961c0047840f3d9: Deleting WAL directory at /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786227706282457-9923-0/minicluster-data/ts-1/wal/wals/db620cc9372649c79a86e438c2e02271
I20260808 22:25:54.123488 7305 ts_tablet_manager.cc:1939] T db620cc9372649c79a86e438c2e02271 P 8f70d639ea46412fb961c0047840f3d9: Deleting consensus metadata
I20260808 22:25:54.124655 6572 catalog_manager.cc:4841] TS 33823b8b2c2844e99803d14a1f12d9fc (127.9.176.195:45445): tablet db620cc9372649c79a86e438c2e02271 (table test-table [id=fd34442726f5468f931f3e8ea92b11ca]) successfully deleted
I20260808 22:25:54.124850 6571 catalog_manager.cc:4841] TS 8f70d639ea46412fb961c0047840f3d9 (127.9.176.194:37235): tablet db620cc9372649c79a86e438c2e02271 (table test-table [id=fd34442726f5468f931f3e8ea92b11ca]) successfully deleted
W20260808 22:25:54.128957 6573 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.176.193:46441 (error 108)
I20260808 22:25:54.129771 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 6876
I20260808 22:25:54.129896 7256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:54.129634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.193:46441 (local address 127.0.0.1:39154)
0808 22:25:54.129751 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:54.129812 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.193:46441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260808 22:25:54.134433 6573 catalog_manager.cc:4584] TS d448ff9563af42e3bf7aaac8f9856c0d (127.9.176.193:46441): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet db620cc9372649c79a86e438c2e02271: Network error: Client connection negotiation failed: client connection to 127.9.176.193:46441: connect: Connection refused (error 111)
I20260808 22:25:54.139560 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 7055
I20260808 22:25:54.150019 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 6544
2026-08-08T22:25:54Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2523 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-08T22:25:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:25:54Z Disabled control of system clock
I20260808 22:25:54.182988 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786227706282457-9923-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786227706282457-9923-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786227706282457-9923-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:46761
--webserver_interface=127.9.176.254
--webserver_port=0
--builtin_ntp_servers=127.9.176.212:40203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.176.254:46761
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 22:25:54.290061 7313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:25:54.290299 7313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:25:54.290330 7313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:25:54.294222 7313 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:25:54.294269 7313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:25:54.294282 7313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:25:54.294299 7313 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:25:54.294329 7313 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:25:54.298939 7313 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:40203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786227706282457-9923-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786227706282457-9923-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.176.254:46761
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:46761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.176.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786227706282457-9923-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:25:54.299862 7313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:25:54.301204 7313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 22:25:54.309201 7313 server_base.cc:1034] running on GCE node
W20260808 22:25:54.309134 7318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:25:54.309149 7321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:25:54.309170 7323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:25:54.309895 7313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:25:54.310956 7313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:25:54.312117 7313 hybrid_clock.cc:648] HybridClock initialized: now 1786227954312109 us; error 29 us; skew 500 ppm
I20260808 22:25:54.313850 7313 webserver.cc:460] Webserver started at http://127.9.176.254:39187/ using document root <none> and password file <none>
I20260808 22:25:54.314363 7313 fs_manager.cc:359] Metadata directory not provided
I20260808 22:25:54.314417 7313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:25:54.314587 7313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:25:54.316512 7313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786227706282457-9923-0/minicluster-data/master-0/data/instance:
uuid: "2a023b17330b415bb0c68603611b5348"
format_stamp: "Formatted at 2026-08-08 22:25:54 on dist-test-slave-ncp9"
server_key: "ba37acf2ffd60a7e621b15b5275953bb"
server_key_iv: "8d66bcbf0485466ca070cdaa12d5d9ee"
server_key_version: "encryptionkey@0"
I20260808 22:25:54.317027 7313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786227706282457-9923-0/minicluster-data/master-0/wal/instance:
uuid: "2a023b17330b415bb0c68603611b5348"
format_stamp: "Formatted at 2026-08-08 22:25:54 on dist-test-slave-ncp9"
server_key: "ba37acf2ffd60a7e621b15b5275953bb"
server_key_iv: "8d66bcbf0485466ca070cdaa12d5d9ee"
server_key_version: "encryptionkey@0"
I20260808 22:25:54.320508 7313 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260808 22:25:54.322825 7335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:25:54.323947 7313 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 22:25:54.324054 7313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786227706282457-9923-0/minicluster-data/master-0/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786227706282457-9923-0/minicluster-data/master-0/wal
uuid: "2a023b17330b415bb0c68603611b5348"
format_stamp: "Formatted at 2026-08-08 22:25:54 on dist-test-slave-ncp9"
server_key: "ba37acf2ffd60a7e621b15b5275953bb"
server_key_iv: "8d66bcbf0485466ca070cdaa12d5d9ee"
server_key_version: "encryptionkey@0"
I20260808 22:25:54.324146 7313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786227706282457-9923-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786227706282457-9923-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786227706282457-9923-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:25:54.337220 7313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:25:54.337769 7313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:25:54.337929 7313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:25:54.344187 7313 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.254:46761
I20260808 22:25:54.344242 7399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.254:46761 every 8 connection(s)
I20260808 22:25:54.345049 7313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
I20260808 22:25:54.347915 7400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:25:54.348172 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 7313
I20260808 22:25:54.348301 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786227706282457-9923-0/minicluster-data/master-0/wal/instance
I20260808 22:25:54.348596 9923 external_mini_cluster.cc:1428] Setting key 901d86d8d5fc205448313f9f0d737991
I20260808 22:25:54.354502 7400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a023b17330b415bb0c68603611b5348: Bootstrap starting.
I20260808 22:25:54.356865 7400 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a023b17330b415bb0c68603611b5348: Neither blocks nor log segments found. Creating new log.
I20260808 22:25:54.357398 7404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:54.349929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41462 (local address 127.9.176.254:46761)
0808 22:25:54.350490 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0808 22:25:54.350505 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 22:25:54.350538 (+ 33us) server_negotiation.cc:407] Connection header received
0808 22:25:54.351213 (+ 675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:25:54.351228 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:25:54.351584 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:25:54.351836 (+ 252us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:25:54.352201 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:54.353178 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:54.354328 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:54.354805 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:54.354915 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:54.355723 (+ 808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:25:54.355728 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:25:54.355739 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:25:54.356085 (+ 346us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:25:54.356148 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:25:54.356167 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:25:54.356237 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0808 22:25:54.356391 (+ 154us) server_negotiation.cc:299] Negotiation successful
0808 22:25:54.356634 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":116,"threads_started":1}
I20260808 22:25:54.357666 7400 log.cc:824] T 00000000000000000000000000000000 P 2a023b17330b415bb0c68603611b5348: Log is configured to *not* fsync() on all Append() calls
I20260808 22:25:54.359516 7400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a023b17330b415bb0c68603611b5348: No bootstrap required, opened a new log
I20260808 22:25:54.362874 7400 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a023b17330b415bb0c68603611b5348 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a023b17330b415bb0c68603611b5348" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 46761 } }
I20260808 22:25:54.363111 7400 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a023b17330b415bb0c68603611b5348 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:25:54.363179 7400 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a023b17330b415bb0c68603611b5348 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a023b17330b415bb0c68603611b5348, State: Initialized, Role: FOLLOWER
I20260808 22:25:54.363559 7400 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a023b17330b415bb0c68603611b5348 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a023b17330b415bb0c68603611b5348" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 46761 } }
I20260808 22:25:54.363675 7400 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a023b17330b415bb0c68603611b5348 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:25:54.363721 7400 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a023b17330b415bb0c68603611b5348 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:25:54.363787 7400 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a023b17330b415bb0c68603611b5348 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:25:54.364571 7400 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a023b17330b415bb0c68603611b5348 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a023b17330b415bb0c68603611b5348" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 46761 } }
I20260808 22:25:54.364869 7400 leader_election.cc:302] T 00000000000000000000000000000000 P 2a023b17330b415bb0c68603611b5348 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a023b17330b415bb0c68603611b5348; no voters:
I20260808 22:25:54.365193 7400 leader_election.cc:288] T 00000000000000000000000000000000 P 2a023b17330b415bb0c68603611b5348 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:25:54.365288 7406 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a023b17330b415bb0c68603611b5348 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:25:54.365485 7406 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a023b17330b415bb0c68603611b5348 [term 1 LEADER]: Becoming Leader. State: Replica: 2a023b17330b415bb0c68603611b5348, State: Running, Role: LEADER
I20260808 22:25:54.365758 7406 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a023b17330b415bb0c68603611b5348 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a023b17330b415bb0c68603611b5348" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 46761 } }
I20260808 22:25:54.366281 7400 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a023b17330b415bb0c68603611b5348 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:25:54.367208 7407 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a023b17330b415bb0c68603611b5348 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a023b17330b415bb0c68603611b5348" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a023b17330b415bb0c68603611b5348" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 46761 } } }
I20260808 22:25:54.367339 7407 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a023b17330b415bb0c68603611b5348 [sys.catalog]: This master's current role is: LEADER
I20260808 22:25:54.367606 7408 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a023b17330b415bb0c68603611b5348 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a023b17330b415bb0c68603611b5348. Latest consensus state: current_term: 1 leader_uuid: "2a023b17330b415bb0c68603611b5348" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a023b17330b415bb0c68603611b5348" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 46761 } } }
I20260808 22:25:54.367707 7408 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a023b17330b415bb0c68603611b5348 [sys.catalog]: This master's current role is: LEADER
I20260808 22:25:54.367761 7420 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:25:54.370496 7420 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:25:54.375836 7420 catalog_manager.cc:1347] Generated new cluster ID: 9fe39296ccfb4b189e8113d3449438ad
I20260808 22:25:54.375891 7420 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:25:54.382156 7420 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:25:54.382993 7420 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:25:54.388398 7420 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a023b17330b415bb0c68603611b5348: Generated new TSK 0
I20260808 22:25:54.389027 7420 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 22:25:54.395452 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786227706282457-9923-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786227706282457-9923-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786227706282457-9923-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.193:0
--local_ip_for_outbound_sockets=127.9.176.193
--webserver_interface=127.9.176.193
--webserver_port=0
--tserver_master_addrs=127.9.176.254:46761
--builtin_ntp_servers=127.9.176.212:40203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 22:25:54.511153 7433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:25:54.511395 7433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:25:54.511432 7433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:25:54.515385 7433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:25:54.515434 7433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:25:54.515497 7433 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.193
I20260808 22:25:54.520109 7433 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:40203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786227706282457-9923-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786227706282457-9923-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.176.193
--webserver_port=0
--tserver_master_addrs=127.9.176.254:46761
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7433
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.193
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786227706282457-9923-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:25:54.521143 7433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:25:54.522473 7433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:25:54.531528 7444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:25:54.531555 7446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:25:54.531708 7433 server_base.cc:1034] running on GCE node
W20260808 22:25:54.531555 7441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:25:54.532173 7433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:25:54.532771 7433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:25:54.533922 7433 hybrid_clock.cc:648] HybridClock initialized: now 1786227954533894 us; error 46 us; skew 500 ppm
I20260808 22:25:54.535597 7433 webserver.cc:460] Webserver started at http://127.9.176.193:35687/ using document root <none> and password file <none>
I20260808 22:25:54.536113 7433 fs_manager.cc:359] Metadata directory not provided
I20260808 22:25:54.536180 7433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:25:54.536346 7433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:25:54.538379 7433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786227706282457-9923-0/minicluster-data/ts-0/data/instance:
uuid: "ac0b06ed388e4ead91c8786ff0f52be5"
format_stamp: "Formatted at 2026-08-08 22:25:54 on dist-test-slave-ncp9"
server_key: "5e0973a021918889becf59d1a178b1c3"
server_key_iv: "8480debb78775246c3375f208e6ce9ce"
server_key_version: "encryptionkey@0"
I20260808 22:25:54.538879 7433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786227706282457-9923-0/minicluster-data/ts-0/wal/instance:
uuid: "ac0b06ed388e4ead91c8786ff0f52be5"
format_stamp: "Formatted at 2026-08-08 22:25:54 on dist-test-slave-ncp9"
server_key: "5e0973a021918889becf59d1a178b1c3"
server_key_iv: "8480debb78775246c3375f208e6ce9ce"
server_key_version: "encryptionkey@0"
I20260808 22:25:54.542449 7433 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260808 22:25:54.544726 7453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:25:54.545950 7433 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 22:25:54.546070 7433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786227706282457-9923-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786227706282457-9923-0/minicluster-data/ts-0/wal
uuid: "ac0b06ed388e4ead91c8786ff0f52be5"
format_stamp: "Formatted at 2026-08-08 22:25:54 on dist-test-slave-ncp9"
server_key: "5e0973a021918889becf59d1a178b1c3"
server_key_iv: "8480debb78775246c3375f208e6ce9ce"
server_key_version: "encryptionkey@0"
I20260808 22:25:54.546161 7433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786227706282457-9923-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786227706282457-9923-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786227706282457-9923-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:25:54.578917 7433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:25:54.579632 7433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:25:54.579818 7433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:25:54.580557 7433 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:25:54.581622 7433 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:25:54.581673 7433 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:25:54.581727 7433 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:25:54.581756 7433 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:25:54.590498 7433 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.193:35059
I20260808 22:25:54.590534 7626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.193:35059 every 8 connection(s)
I20260808 22:25:54.591401 7433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786227706282457-9923-0/minicluster-data/ts-0/data/info.pb
I20260808 22:25:54.592983 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 7433
I20260808 22:25:54.593104 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786227706282457-9923-0/minicluster-data/ts-0/wal/instance
I20260808 22:25:54.593421 9923 external_mini_cluster.cc:1428] Setting key 7423598a0bbba2a394e573fb8b529be9
I20260808 22:25:54.595252 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786227706282457-9923-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786227706282457-9923-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786227706282457-9923-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.194:0
--local_ip_for_outbound_sockets=127.9.176.194
--webserver_interface=127.9.176.194
--webserver_port=0
--tserver_master_addrs=127.9.176.254:46761
--builtin_ntp_servers=127.9.176.212:40203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 22:25:54.600419 7404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:54.593150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.193:51345 (local address 127.9.176.254:46761)
0808 22:25:54.593301 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0808 22:25:54.593307 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 22:25:54.594468 (+ 1161us) server_negotiation.cc:407] Connection header received
0808 22:25:54.595868 (+ 1400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:25:54.595871 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:25:54.595923 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:25:54.595995 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:25:54.596878 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:54.597470 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:54.598626 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:54.598850 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:54.598939 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:54.599961 (+ 1022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:25:54.599964 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:25:54.599966 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:25:54.599978 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:25:54.599997 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:25:54.600005 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:25:54.600062 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0808 22:25:54.600209 (+ 147us) server_negotiation.cc:299] Negotiation successful
0808 22:25:54.600275 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 22:25:54.601306 7629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:54.593560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:46761 (local address 127.9.176.193:51345)
0808 22:25:54.594314 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:54.594366 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 22:25:54.595670 (+ 1304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:25:54.596178 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:25:54.596184 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:25:54.596269 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:25:54.596747 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:54.596760 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:54.597597 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:54.597600 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:54.598503 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:54.598513 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:54.598997 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:54.599001 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:54.599159 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:54.599714 (+ 555us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 22:25:54.599715 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:25:54.599756 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 22:25:54.599820 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:25:54.600073 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:25:54.600083 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:25:54.600095 (+ 12us) client_negotiation.cc:769] Sending connection context
0808 22:25:54.600201 (+ 106us) client_negotiation.cc:240] Negotiation successful
0808 22:25:54.600483 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":115,"threads_started":1}
I20260808 22:25:54.602492 7627 heartbeater.cc:347] Connected to a master server at 127.9.176.254:46761
I20260808 22:25:54.602772 7627 heartbeater.cc:464] Registering TS with master...
I20260808 22:25:54.603423 7627 heartbeater.cc:511] Master 127.9.176.254:46761 requested a full tablet report, sending...
I20260808 22:25:54.605016 7356 ts_manager.cc:194] Registered new tserver with Master: ac0b06ed388e4ead91c8786ff0f52be5 (127.9.176.193:35059)
I20260808 22:25:54.606117 7356 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.176.193:51345
W20260808 22:25:54.709249 7630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:25:54.709582 7630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:25:54.709653 7630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:25:54.714764 7630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:25:54.714829 7630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:25:54.714910 7630 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.194
I20260808 22:25:54.720566 7630 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:40203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786227706282457-9923-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786227706282457-9923-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.176.194
--webserver_port=0
--tserver_master_addrs=127.9.176.254:46761
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7630
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.194
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786227706282457-9923-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:25:54.722018 7630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:25:54.723919 7630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:25:54.735235 7641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:25:54.735227 7639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:25:54.735225 7638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:25:54.738116 7630 server_base.cc:1034] running on GCE node
I20260808 22:25:54.738509 7630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:25:54.739115 7630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:25:54.746428 7630 hybrid_clock.cc:648] HybridClock initialized: now 1786227954746392 us; error 59 us; skew 500 ppm
I20260808 22:25:54.748458 7630 webserver.cc:460] Webserver started at http://127.9.176.194:36167/ using document root <none> and password file <none>
I20260808 22:25:54.749185 7630 fs_manager.cc:359] Metadata directory not provided
I20260808 22:25:54.749269 7630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:25:54.749492 7630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:25:54.751953 7630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786227706282457-9923-0/minicluster-data/ts-1/data/instance:
uuid: "ad6134eec77848e4887b3dec3b1c8bbb"
format_stamp: "Formatted at 2026-08-08 22:25:54 on dist-test-slave-ncp9"
server_key: "961a63c0b0c8c46b8704891eac0bc571"
server_key_iv: "33fbbcde3ea6ef6c188d6d1b4f0d5ba3"
server_key_version: "encryptionkey@0"
I20260808 22:25:54.752570 7630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786227706282457-9923-0/minicluster-data/ts-1/wal/instance:
uuid: "ad6134eec77848e4887b3dec3b1c8bbb"
format_stamp: "Formatted at 2026-08-08 22:25:54 on dist-test-slave-ncp9"
server_key: "961a63c0b0c8c46b8704891eac0bc571"
server_key_iv: "33fbbcde3ea6ef6c188d6d1b4f0d5ba3"
server_key_version: "encryptionkey@0"
I20260808 22:25:54.756204 7630 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260808 22:25:54.758548 7652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:25:54.759660 7630 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 22:25:54.759768 7630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786227706282457-9923-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786227706282457-9923-0/minicluster-data/ts-1/wal
uuid: "ad6134eec77848e4887b3dec3b1c8bbb"
format_stamp: "Formatted at 2026-08-08 22:25:54 on dist-test-slave-ncp9"
server_key: "961a63c0b0c8c46b8704891eac0bc571"
server_key_iv: "33fbbcde3ea6ef6c188d6d1b4f0d5ba3"
server_key_version: "encryptionkey@0"
I20260808 22:25:54.759861 7630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786227706282457-9923-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786227706282457-9923-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786227706282457-9923-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:25:54.774421 7630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:25:54.775041 7630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:25:54.775209 7630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:25:54.775761 7630 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:25:54.776655 7630 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:25:54.776702 7630 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:25:54.776754 7630 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:25:54.776783 7630 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:25:54.784857 7630 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.194:33613
I20260808 22:25:54.784950 7810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.194:33613 every 8 connection(s)
I20260808 22:25:54.785739 7630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786227706282457-9923-0/minicluster-data/ts-1/data/info.pb
I20260808 22:25:54.791975 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 7630
I20260808 22:25:54.792097 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786227706282457-9923-0/minicluster-data/ts-1/wal/instance
I20260808 22:25:54.792402 9923 external_mini_cluster.cc:1428] Setting key bc3049ea9ae2ee41ad2ea3348621ef5b
I20260808 22:25:54.794364 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786227706282457-9923-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786227706282457-9923-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786227706282457-9923-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.176.195:0
--local_ip_for_outbound_sockets=127.9.176.195
--webserver_interface=127.9.176.195
--webserver_port=0
--tserver_master_addrs=127.9.176.254:46761
--builtin_ntp_servers=127.9.176.212:40203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 22:25:54.794699 7404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:54.787457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.194:53891 (local address 127.9.176.254:46761)
0808 22:25:54.787590 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0808 22:25:54.787593 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:25:54.788605 (+ 1012us) server_negotiation.cc:407] Connection header received
0808 22:25:54.789981 (+ 1376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:25:54.789983 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:25:54.790031 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:25:54.790104 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:25:54.790924 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:54.791438 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:54.792872 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:54.793156 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:54.793249 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:54.794256 (+ 1007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:25:54.794259 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:25:54.794261 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:25:54.794271 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:25:54.794291 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:25:54.794299 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:25:54.794355 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0808 22:25:54.794534 (+ 179us) server_negotiation.cc:299] Negotiation successful
0808 22:25:54.794590 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 22:25:54.795768 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:54.787814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:46761 (local address 127.9.176.194:53891)
0808 22:25:54.788454 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:54.788494 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 22:25:54.789795 (+ 1301us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:25:54.790267 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:25:54.790273 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:25:54.790355 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:25:54.790818 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:54.790830 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:54.791519 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:54.791521 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:54.792749 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:54.792762 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:54.793300 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:54.793305 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:54.793453 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:54.794008 (+ 555us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 22:25:54.794010 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:25:54.794048 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 22:25:54.794116 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:25:54.794366 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:25:54.794375 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:25:54.794381 (+ 6us) client_negotiation.cc:769] Sending connection context
0808 22:25:54.794851 (+ 470us) client_negotiation.cc:240] Negotiation successful
0808 22:25:54.795080 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":116,"threads_started":1}
I20260808 22:25:54.796808 7811 heartbeater.cc:347] Connected to a master server at 127.9.176.254:46761
I20260808 22:25:54.797075 7811 heartbeater.cc:464] Registering TS with master...
I20260808 22:25:54.797559 7811 heartbeater.cc:511] Master 127.9.176.254:46761 requested a full tablet report, sending...
I20260808 22:25:54.798451 7356 ts_manager.cc:194] Registered new tserver with Master: ad6134eec77848e4887b3dec3b1c8bbb (127.9.176.194:33613)
I20260808 22:25:54.798980 7356 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.176.194:53891
W20260808 22:25:54.906541 7815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:25:54.906757 7815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:25:54.906795 7815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:25:54.910737 7815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:25:54.910787 7815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:25:54.910851 7815 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.176.195
I20260808 22:25:54.915426 7815 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:40203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786227706282457-9923-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786227706282457-9923-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.176.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.176.195
--webserver_port=0
--tserver_master_addrs=127.9.176.254:46761
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7815
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.176.195
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786227706282457-9923-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:25:54.916348 7815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:25:54.917657 7815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:25:54.927021 7826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:25:54.927079 7822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:25:54.927192 7823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:25:54.927261 7815 server_base.cc:1034] running on GCE node
I20260808 22:25:54.927726 7815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:25:54.928331 7815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:25:54.929498 7815 hybrid_clock.cc:648] HybridClock initialized: now 1786227954929486 us; error 29 us; skew 500 ppm
I20260808 22:25:54.934172 7815 webserver.cc:460] Webserver started at http://127.9.176.195:35751/ using document root <none> and password file <none>
I20260808 22:25:54.934715 7815 fs_manager.cc:359] Metadata directory not provided
I20260808 22:25:54.934779 7815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:25:54.934978 7815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:25:54.937047 7815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786227706282457-9923-0/minicluster-data/ts-2/data/instance:
uuid: "7a28325db3974947bc5a325e1d9286c6"
format_stamp: "Formatted at 2026-08-08 22:25:54 on dist-test-slave-ncp9"
server_key: "29dcdc8ee2a00a09e8a6721733b5157c"
server_key_iv: "77211a8e8ef41e57fedb04039acf821d"
server_key_version: "encryptionkey@0"
I20260808 22:25:54.937589 7815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786227706282457-9923-0/minicluster-data/ts-2/wal/instance:
uuid: "7a28325db3974947bc5a325e1d9286c6"
format_stamp: "Formatted at 2026-08-08 22:25:54 on dist-test-slave-ncp9"
server_key: "29dcdc8ee2a00a09e8a6721733b5157c"
server_key_iv: "77211a8e8ef41e57fedb04039acf821d"
server_key_version: "encryptionkey@0"
I20260808 22:25:54.941052 7815 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260808 22:25:54.943269 7835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:25:54.944370 7815 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 22:25:54.944473 7815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786227706282457-9923-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786227706282457-9923-0/minicluster-data/ts-2/wal
uuid: "7a28325db3974947bc5a325e1d9286c6"
format_stamp: "Formatted at 2026-08-08 22:25:54 on dist-test-slave-ncp9"
server_key: "29dcdc8ee2a00a09e8a6721733b5157c"
server_key_iv: "77211a8e8ef41e57fedb04039acf821d"
server_key_version: "encryptionkey@0"
I20260808 22:25:54.944561 7815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786227706282457-9923-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786227706282457-9923-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786227706282457-9923-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:25:54.955825 7815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:25:54.956394 7815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:25:54.956550 7815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:25:54.957145 7815 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:25:54.958011 7815 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:25:54.958058 7815 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:25:54.958117 7815 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:25:54.958141 7815 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:25:54.966548 7815 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.195:42541
I20260808 22:25:54.966609 7987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.195:42541 every 8 connection(s)
I20260808 22:25:54.967450 7815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786227706282457-9923-0/minicluster-data/ts-2/data/info.pb
I20260808 22:25:54.971843 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 7815
I20260808 22:25:54.971961 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786227706282457-9923-0/minicluster-data/ts-2/wal/instance
I20260808 22:25:54.972224 9923 external_mini_cluster.cc:1428] Setting key 03f6f6a4c88a2023c28c583d199f3f56
I20260808 22:25:54.976341 7404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:54.969213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.195:46439 (local address 127.9.176.254:46761)
0808 22:25:54.969370 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0808 22:25:54.969375 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:25:54.970401 (+ 1026us) server_negotiation.cc:407] Connection header received
0808 22:25:54.971738 (+ 1337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:25:54.971742 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:25:54.971799 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:25:54.971887 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:25:54.972809 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:54.973519 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:54.974721 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:54.974939 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:54.975011 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:54.976001 (+ 990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:25:54.976004 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:25:54.976006 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:25:54.976017 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:25:54.976034 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:25:54.976041 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:25:54.976083 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0808 22:25:54.976188 (+ 105us) server_negotiation.cc:299] Negotiation successful
0808 22:25:54.976233 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 22:25:54.977193 7990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:54.969615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:46761 (local address 127.9.176.195:46439)
0808 22:25:54.970257 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:54.970296 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 22:25:54.971527 (+ 1231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:25:54.972052 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:25:54.972059 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:25:54.972151 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:25:54.972658 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:54.972677 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:54.973638 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:54.973641 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:54.974606 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:54.974617 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:54.975045 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:54.975047 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:54.975194 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:54.975774 (+ 580us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 22:25:54.975776 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:25:54.975816 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 22:25:54.975878 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:25:54.976089 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:25:54.976093 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:25:54.976096 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 22:25:54.976183 (+ 87us) client_negotiation.cc:240] Negotiation successful
0808 22:25:54.976395 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":117,"threads_started":1}
I20260808 22:25:54.978199 7988 heartbeater.cc:347] Connected to a master server at 127.9.176.254:46761
I20260808 22:25:54.978407 7988 heartbeater.cc:464] Registering TS with master...
I20260808 22:25:54.978889 7988 heartbeater.cc:511] Master 127.9.176.254:46761 requested a full tablet report, sending...
I20260808 22:25:54.979828 7356 ts_manager.cc:194] Registered new tserver with Master: 7a28325db3974947bc5a325e1d9286c6 (127.9.176.195:42541)
I20260808 22:25:54.980376 7356 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.176.195:46439
I20260808 22:25:54.985764 9923 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 22:25:54.990788 7404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:54.987642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41476 (local address 127.9.176.254:46761)
0808 22:25:54.987777 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0808 22:25:54.987781 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:25:54.987905 (+ 124us) server_negotiation.cc:407] Connection header received
0808 22:25:54.987995 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:25:54.987997 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:25:54.988034 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:25:54.988090 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:25:54.988299 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:54.988781 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:54.989532 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:54.989741 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:54.989810 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:54.990472 (+ 662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:25:54.990474 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:25:54.990476 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:25:54.990485 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:25:54.990498 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:25:54.990505 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:25:54.990542 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0808 22:25:54.990615 (+ 73us) server_negotiation.cc:299] Negotiation successful
0808 22:25:54.990656 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 22:25:54.993359 7356 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41476:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 22:25:54.996249 7356 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 22:25:55.010875 8001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:55.004168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.193:35059 (local address 127.0.0.1:59980)
0808 22:25:55.004640 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:55.004687 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0808 22:25:55.004808 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:25:55.006277 (+ 1469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:25:55.006309 (+ 32us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:25:55.006335 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:25:55.006472 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:55.006479 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:55.008011 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:55.008014 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:55.009292 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:55.009303 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:55.010321 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:55.010325 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:55.010436 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:55.010475 (+ 39us) client_negotiation.cc:769] Sending connection context
0808 22:25:55.010602 (+ 127us) client_negotiation.cc:240] Negotiation successful
0808 22:25:55.010720 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"mutex_wait_us":69,"thread_start_us":82,"threads_started":1}
I20260808 22:25:55.010900 8004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:55.004300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59980 (local address 127.9.176.193:35059)
0808 22:25:55.005326 (+ 1026us) server_negotiation.cc:206] Beginning negotiation
0808 22:25:55.005339 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 22:25:55.005367 (+ 28us) server_negotiation.cc:407] Connection header received
0808 22:25:55.005866 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:25:55.005872 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:25:55.006101 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:25:55.006257 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:25:55.006591 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:55.007835 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:55.009416 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:55.010191 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:55.010307 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:55.010518 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0808 22:25:55.010628 (+ 110us) server_negotiation.cc:299] Negotiation successful
0808 22:25:55.010730 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":805,"thread_start_us":148,"threads_started":1}
I20260808 22:25:55.011219 8008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:55.004979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48454 (local address 127.9.176.194:33613)
0808 22:25:55.005279 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0808 22:25:55.005291 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 22:25:55.005702 (+ 411us) server_negotiation.cc:407] Connection header received
0808 22:25:55.005795 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:25:55.005801 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:25:55.006163 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:25:55.006315 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:25:55.006651 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:55.007835 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:55.009720 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:55.010263 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:55.010362 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:55.010548 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0808 22:25:55.010945 (+ 397us) server_negotiation.cc:299] Negotiation successful
0808 22:25:55.011052 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":89,"threads_started":1}
I20260808 22:25:55.011230 8006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:55.004295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.194:33613 (local address 127.0.0.1:48454)
0808 22:25:55.005459 (+ 1164us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:55.005472 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 22:25:55.005562 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:25:55.006432 (+ 870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:25:55.006435 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:25:55.006453 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:25:55.006548 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:55.006554 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:55.007960 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:55.007963 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:55.009563 (+ 1600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:55.009574 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:55.010609 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:55.010613 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:55.010694 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:55.010714 (+ 20us) client_negotiation.cc:769] Sending connection context
0808 22:25:55.010777 (+ 63us) client_negotiation.cc:240] Negotiation successful
0808 22:25:55.010831 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1102,"mutex_wait_us":88,"thread_start_us":98,"threads_started":1}
I20260808 22:25:55.011734 8003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:55.004787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.195:42541 (local address 127.0.0.1:57610)
0808 22:25:55.005511 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:55.005524 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 22:25:55.005613 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:25:55.006893 (+ 1280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:25:55.006896 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:25:55.006912 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:25:55.007004 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:55.007010 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:55.007969 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:55.007972 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:55.009563 (+ 1591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:55.009574 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:55.010781 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:55.010785 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:55.010872 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:55.010897 (+ 25us) client_negotiation.cc:769] Sending connection context
0808 22:25:55.011543 (+ 646us) client_negotiation.cc:240] Negotiation successful
0808 22:25:55.011620 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":656,"mutex_wait_us":252,"thread_start_us":131,"threads_started":1}
I20260808 22:25:55.013366 8009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:55.004982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57610 (local address 127.9.176.195:42541)
0808 22:25:55.006511 (+ 1529us) server_negotiation.cc:206] Beginning negotiation
0808 22:25:55.006523 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 22:25:55.006545 (+ 22us) server_negotiation.cc:407] Connection header received
0808 22:25:55.006630 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:25:55.006636 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:25:55.006772 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:25:55.006892 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:25:55.007107 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:55.007854 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:55.009694 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:55.010441 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:55.010540 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:55.010717 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0808 22:25:55.013107 (+ 2390us) server_negotiation.cc:299] Negotiation successful
0808 22:25:55.013212 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1433,"thread_start_us":142,"threads_started":1}
I20260808 22:25:55.013855 7719 tablet_service.cc:1467] Processing CreateTablet for tablet 907e1a5f97a94e9d8021eb6e009f97db (DEFAULT_TABLE table=test-table [id=ccaba8ba6bda4e11ac7f742894f62a9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:25:55.013885 7521 tablet_service.cc:1467] Processing CreateTablet for tablet 907e1a5f97a94e9d8021eb6e009f97db (DEFAULT_TABLE table=test-table [id=ccaba8ba6bda4e11ac7f742894f62a9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:25:55.014837 7521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 907e1a5f97a94e9d8021eb6e009f97db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:25:55.014883 7719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 907e1a5f97a94e9d8021eb6e009f97db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:25:55.015945 7893 tablet_service.cc:1467] Processing CreateTablet for tablet 907e1a5f97a94e9d8021eb6e009f97db (DEFAULT_TABLE table=test-table [id=ccaba8ba6bda4e11ac7f742894f62a9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:25:55.016949 7893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 907e1a5f97a94e9d8021eb6e009f97db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:25:55.020928 8011 tablet_bootstrap.cc:492] T 907e1a5f97a94e9d8021eb6e009f97db P ac0b06ed388e4ead91c8786ff0f52be5: Bootstrap starting.
I20260808 22:25:55.022686 8012 tablet_bootstrap.cc:492] T 907e1a5f97a94e9d8021eb6e009f97db P ad6134eec77848e4887b3dec3b1c8bbb: Bootstrap starting.
I20260808 22:25:55.023065 8011 tablet_bootstrap.cc:654] T 907e1a5f97a94e9d8021eb6e009f97db P ac0b06ed388e4ead91c8786ff0f52be5: Neither blocks nor log segments found. Creating new log.
I20260808 22:25:55.023805 8014 tablet_bootstrap.cc:492] T 907e1a5f97a94e9d8021eb6e009f97db P 7a28325db3974947bc5a325e1d9286c6: Bootstrap starting.
I20260808 22:25:55.024088 8011 log.cc:824] T 907e1a5f97a94e9d8021eb6e009f97db P ac0b06ed388e4ead91c8786ff0f52be5: Log is configured to *not* fsync() on all Append() calls
I20260808 22:25:55.024766 8012 tablet_bootstrap.cc:654] T 907e1a5f97a94e9d8021eb6e009f97db P ad6134eec77848e4887b3dec3b1c8bbb: Neither blocks nor log segments found. Creating new log.
I20260808 22:25:55.025629 8012 log.cc:824] T 907e1a5f97a94e9d8021eb6e009f97db P ad6134eec77848e4887b3dec3b1c8bbb: Log is configured to *not* fsync() on all Append() calls
I20260808 22:25:55.025764 8014 tablet_bootstrap.cc:654] T 907e1a5f97a94e9d8021eb6e009f97db P 7a28325db3974947bc5a325e1d9286c6: Neither blocks nor log segments found. Creating new log.
I20260808 22:25:55.026486 8011 tablet_bootstrap.cc:492] T 907e1a5f97a94e9d8021eb6e009f97db P ac0b06ed388e4ead91c8786ff0f52be5: No bootstrap required, opened a new log
I20260808 22:25:55.026518 8014 log.cc:824] T 907e1a5f97a94e9d8021eb6e009f97db P 7a28325db3974947bc5a325e1d9286c6: Log is configured to *not* fsync() on all Append() calls
I20260808 22:25:55.026687 8011 ts_tablet_manager.cc:1397] T 907e1a5f97a94e9d8021eb6e009f97db P ac0b06ed388e4ead91c8786ff0f52be5: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 22:25:55.027654 8012 tablet_bootstrap.cc:492] T 907e1a5f97a94e9d8021eb6e009f97db P ad6134eec77848e4887b3dec3b1c8bbb: No bootstrap required, opened a new log
I20260808 22:25:55.027840 8012 ts_tablet_manager.cc:1397] T 907e1a5f97a94e9d8021eb6e009f97db P ad6134eec77848e4887b3dec3b1c8bbb: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260808 22:25:55.028659 8014 tablet_bootstrap.cc:492] T 907e1a5f97a94e9d8021eb6e009f97db P 7a28325db3974947bc5a325e1d9286c6: No bootstrap required, opened a new log
I20260808 22:25:55.028837 8014 ts_tablet_manager.cc:1397] T 907e1a5f97a94e9d8021eb6e009f97db P 7a28325db3974947bc5a325e1d9286c6: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260808 22:25:55.030164 8011 raft_consensus.cc:348] T 907e1a5f97a94e9d8021eb6e009f97db P ac0b06ed388e4ead91c8786ff0f52be5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac0b06ed388e4ead91c8786ff0f52be5" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35059 } } peers { permanent_uuid: "ad6134eec77848e4887b3dec3b1c8bbb" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 33613 } } peers { permanent_uuid: "7a28325db3974947bc5a325e1d9286c6" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 42541 } }
I20260808 22:25:55.030385 8011 raft_consensus.cc:374] T 907e1a5f97a94e9d8021eb6e009f97db P ac0b06ed388e4ead91c8786ff0f52be5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:25:55.030436 8011 raft_consensus.cc:729] T 907e1a5f97a94e9d8021eb6e009f97db P ac0b06ed388e4ead91c8786ff0f52be5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac0b06ed388e4ead91c8786ff0f52be5, State: Initialized, Role: FOLLOWER
I20260808 22:25:55.030855 8011 consensus_queue.cc:260] T 907e1a5f97a94e9d8021eb6e009f97db P ac0b06ed388e4ead91c8786ff0f52be5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac0b06ed388e4ead91c8786ff0f52be5" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35059 } } peers { permanent_uuid: "ad6134eec77848e4887b3dec3b1c8bbb" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 33613 } } peers { permanent_uuid: "7a28325db3974947bc5a325e1d9286c6" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 42541 } }
I20260808 22:25:55.031505 7627 heartbeater.cc:503] Master 127.9.176.254:46761 was elected leader, sending a full tablet report...
I20260808 22:25:55.031854 8011 ts_tablet_manager.cc:1428] T 907e1a5f97a94e9d8021eb6e009f97db P ac0b06ed388e4ead91c8786ff0f52be5: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260808 22:25:55.032028 8012 raft_consensus.cc:348] T 907e1a5f97a94e9d8021eb6e009f97db P ad6134eec77848e4887b3dec3b1c8bbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac0b06ed388e4ead91c8786ff0f52be5" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35059 } } peers { permanent_uuid: "ad6134eec77848e4887b3dec3b1c8bbb" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 33613 } } peers { permanent_uuid: "7a28325db3974947bc5a325e1d9286c6" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 42541 } }
I20260808 22:25:55.032514 8012 raft_consensus.cc:374] T 907e1a5f97a94e9d8021eb6e009f97db P ad6134eec77848e4887b3dec3b1c8bbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:25:55.032568 8012 raft_consensus.cc:729] T 907e1a5f97a94e9d8021eb6e009f97db P ad6134eec77848e4887b3dec3b1c8bbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad6134eec77848e4887b3dec3b1c8bbb, State: Initialized, Role: FOLLOWER
I20260808 22:25:55.032856 8014 raft_consensus.cc:348] T 907e1a5f97a94e9d8021eb6e009f97db P 7a28325db3974947bc5a325e1d9286c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac0b06ed388e4ead91c8786ff0f52be5" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35059 } } peers { permanent_uuid: "ad6134eec77848e4887b3dec3b1c8bbb" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 33613 } } peers { permanent_uuid: "7a28325db3974947bc5a325e1d9286c6" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 42541 } }
I20260808 22:25:55.033113 8014 raft_consensus.cc:374] T 907e1a5f97a94e9d8021eb6e009f97db P 7a28325db3974947bc5a325e1d9286c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:25:55.033052 8012 consensus_queue.cc:260] T 907e1a5f97a94e9d8021eb6e009f97db P ad6134eec77848e4887b3dec3b1c8bbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac0b06ed388e4ead91c8786ff0f52be5" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35059 } } peers { permanent_uuid: "ad6134eec77848e4887b3dec3b1c8bbb" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 33613 } } peers { permanent_uuid: "7a28325db3974947bc5a325e1d9286c6" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 42541 } }
I20260808 22:25:55.033155 8014 raft_consensus.cc:729] T 907e1a5f97a94e9d8021eb6e009f97db P 7a28325db3974947bc5a325e1d9286c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a28325db3974947bc5a325e1d9286c6, State: Initialized, Role: FOLLOWER
I20260808 22:25:55.033556 8014 consensus_queue.cc:260] T 907e1a5f97a94e9d8021eb6e009f97db P 7a28325db3974947bc5a325e1d9286c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac0b06ed388e4ead91c8786ff0f52be5" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35059 } } peers { permanent_uuid: "ad6134eec77848e4887b3dec3b1c8bbb" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 33613 } } peers { permanent_uuid: "7a28325db3974947bc5a325e1d9286c6" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 42541 } }
I20260808 22:25:55.033797 7811 heartbeater.cc:503] Master 127.9.176.254:46761 was elected leader, sending a full tablet report...
I20260808 22:25:55.034119 8012 ts_tablet_manager.cc:1428] T 907e1a5f97a94e9d8021eb6e009f97db P ad6134eec77848e4887b3dec3b1c8bbb: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260808 22:25:55.034409 8014 ts_tablet_manager.cc:1428] T 907e1a5f97a94e9d8021eb6e009f97db P 7a28325db3974947bc5a325e1d9286c6: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260808 22:25:55.034716 7988 heartbeater.cc:503] Master 127.9.176.254:46761 was elected leader, sending a full tablet report...
W20260808 22:25:55.036995 7812 tablet.cc:2369] T 907e1a5f97a94e9d8021eb6e009f97db P ad6134eec77848e4887b3dec3b1c8bbb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 22:25:55.092551 7628 tablet.cc:2369] T 907e1a5f97a94e9d8021eb6e009f97db P ac0b06ed388e4ead91c8786ff0f52be5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 22:25:55.179872 8024 raft_consensus.cc:482] T 907e1a5f97a94e9d8021eb6e009f97db P 7a28325db3974947bc5a325e1d9286c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 22:25:55.180080 8024 raft_consensus.cc:504] T 907e1a5f97a94e9d8021eb6e009f97db P 7a28325db3974947bc5a325e1d9286c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac0b06ed388e4ead91c8786ff0f52be5" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35059 } } peers { permanent_uuid: "ad6134eec77848e4887b3dec3b1c8bbb" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 33613 } } peers { permanent_uuid: "7a28325db3974947bc5a325e1d9286c6" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 42541 } }
I20260808 22:25:55.181239 8024 leader_election.cc:288] T 907e1a5f97a94e9d8021eb6e009f97db P 7a28325db3974947bc5a325e1d9286c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac0b06ed388e4ead91c8786ff0f52be5 (127.9.176.193:35059), ad6134eec77848e4887b3dec3b1c8bbb (127.9.176.194:33613)
I20260808 22:25:55.185159 7990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:55.181484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.193:35059 (local address 127.9.176.195:37053)
0808 22:25:55.181654 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:55.181670 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 22:25:55.181769 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:25:55.182061 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:25:55.182064 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:25:55.182081 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:25:55.182219 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:55.182229 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:55.183161 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:55.183164 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:55.183959 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:55.183968 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:55.184792 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:55.184796 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:55.184884 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:55.184925 (+ 41us) client_negotiation.cc:769] Sending connection context
0808 22:25:55.184987 (+ 62us) client_negotiation.cc:240] Negotiation successful
0808 22:25:55.185042 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260808 22:25:55.185174 8004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:55.181577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.195:37053 (local address 127.9.176.193:35059)
0808 22:25:55.181713 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0808 22:25:55.181717 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:25:55.181736 (+ 19us) server_negotiation.cc:407] Connection header received
0808 22:25:55.181913 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:25:55.181916 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:25:55.181965 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:25:55.182027 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:25:55.182354 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:55.183056 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:55.184078 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:55.184475 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:55.184605 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:55.184741 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0808 22:25:55.185005 (+ 264us) server_negotiation.cc:299] Negotiation successful
0808 22:25:55.185073 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 22:25:55.185174 8008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:55.181699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.176.195:41937 (local address 127.9.176.194:33613)
0808 22:25:55.181819 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0808 22:25:55.181823 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:25:55.182025 (+ 202us) server_negotiation.cc:407] Connection header received
0808 22:25:55.182153 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:25:55.182155 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:25:55.182211 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:25:55.182272 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:25:55.182527 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:55.183083 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:55.184106 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:55.184477 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:55.184580 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:55.184631 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0808 22:25:55.184953 (+ 322us) server_negotiation.cc:299] Negotiation successful
0808 22:25:55.185039 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 22:25:55.185171 8025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:55.181501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.194:33613 (local address 127.9.176.195:41937)
0808 22:25:55.181933 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:55.181943 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 22:25:55.182045 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:25:55.182292 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:25:55.182295 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:25:55.182310 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:25:55.182426 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:55.182431 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:55.183197 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:55.183201 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:55.183981 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:25:55.183989 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:25:55.184580 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:25:55.184584 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:25:55.184798 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:55.184823 (+ 25us) client_negotiation.cc:769] Sending connection context
0808 22:25:55.184925 (+ 102us) client_negotiation.cc:240] Negotiation successful
0808 22:25:55.184995 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"mutex_wait_us":1,"thread_start_us":131,"threads_started":1}
I20260808 22:25:55.185952 7548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "907e1a5f97a94e9d8021eb6e009f97db" candidate_uuid: "7a28325db3974947bc5a325e1d9286c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac0b06ed388e4ead91c8786ff0f52be5" is_pre_election: true
I20260808 22:25:55.186023 7745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "907e1a5f97a94e9d8021eb6e009f97db" candidate_uuid: "7a28325db3974947bc5a325e1d9286c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad6134eec77848e4887b3dec3b1c8bbb" is_pre_election: true
I20260808 22:25:55.186301 7548 raft_consensus.cc:2454] T 907e1a5f97a94e9d8021eb6e009f97db P ac0b06ed388e4ead91c8786ff0f52be5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a28325db3974947bc5a325e1d9286c6 in term 0.
I20260808 22:25:55.186297 7745 raft_consensus.cc:2454] T 907e1a5f97a94e9d8021eb6e009f97db P ad6134eec77848e4887b3dec3b1c8bbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a28325db3974947bc5a325e1d9286c6 in term 0.
I20260808 22:25:55.186822 7839 leader_election.cc:302] T 907e1a5f97a94e9d8021eb6e009f97db P 7a28325db3974947bc5a325e1d9286c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a28325db3974947bc5a325e1d9286c6, ad6134eec77848e4887b3dec3b1c8bbb; no voters:
I20260808 22:25:55.187098 8024 raft_consensus.cc:2781] T 907e1a5f97a94e9d8021eb6e009f97db P 7a28325db3974947bc5a325e1d9286c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 22:25:55.187183 8024 raft_consensus.cc:482] T 907e1a5f97a94e9d8021eb6e009f97db P 7a28325db3974947bc5a325e1d9286c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 22:25:55.187234 8024 raft_consensus.cc:3037] T 907e1a5f97a94e9d8021eb6e009f97db P 7a28325db3974947bc5a325e1d9286c6 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:25:55.188304 8024 raft_consensus.cc:504] T 907e1a5f97a94e9d8021eb6e009f97db P 7a28325db3974947bc5a325e1d9286c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac0b06ed388e4ead91c8786ff0f52be5" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35059 } } peers { permanent_uuid: "ad6134eec77848e4887b3dec3b1c8bbb" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 33613 } } peers { permanent_uuid: "7a28325db3974947bc5a325e1d9286c6" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 42541 } }
I20260808 22:25:55.188623 8024 leader_election.cc:288] T 907e1a5f97a94e9d8021eb6e009f97db P 7a28325db3974947bc5a325e1d9286c6 [CANDIDATE]: Term 1 election: Requested vote from peers ac0b06ed388e4ead91c8786ff0f52be5 (127.9.176.193:35059), ad6134eec77848e4887b3dec3b1c8bbb (127.9.176.194:33613)
I20260808 22:25:55.188948 7548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "907e1a5f97a94e9d8021eb6e009f97db" candidate_uuid: "7a28325db3974947bc5a325e1d9286c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac0b06ed388e4ead91c8786ff0f52be5"
I20260808 22:25:55.189004 7745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "907e1a5f97a94e9d8021eb6e009f97db" candidate_uuid: "7a28325db3974947bc5a325e1d9286c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad6134eec77848e4887b3dec3b1c8bbb"
I20260808 22:25:55.189057 7548 raft_consensus.cc:3037] T 907e1a5f97a94e9d8021eb6e009f97db P ac0b06ed388e4ead91c8786ff0f52be5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:25:55.189136 7745 raft_consensus.cc:3037] T 907e1a5f97a94e9d8021eb6e009f97db P ad6134eec77848e4887b3dec3b1c8bbb [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:25:55.190227 7548 raft_consensus.cc:2454] T 907e1a5f97a94e9d8021eb6e009f97db P ac0b06ed388e4ead91c8786ff0f52be5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a28325db3974947bc5a325e1d9286c6 in term 1.
I20260808 22:25:55.190239 7745 raft_consensus.cc:2454] T 907e1a5f97a94e9d8021eb6e009f97db P ad6134eec77848e4887b3dec3b1c8bbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a28325db3974947bc5a325e1d9286c6 in term 1.
I20260808 22:25:55.190518 7838 leader_election.cc:302] T 907e1a5f97a94e9d8021eb6e009f97db P 7a28325db3974947bc5a325e1d9286c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a28325db3974947bc5a325e1d9286c6, ac0b06ed388e4ead91c8786ff0f52be5; no voters:
I20260808 22:25:55.190706 8024 raft_consensus.cc:2781] T 907e1a5f97a94e9d8021eb6e009f97db P 7a28325db3974947bc5a325e1d9286c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:25:55.190900 8024 raft_consensus.cc:686] T 907e1a5f97a94e9d8021eb6e009f97db P 7a28325db3974947bc5a325e1d9286c6 [term 1 LEADER]: Becoming Leader. State: Replica: 7a28325db3974947bc5a325e1d9286c6, State: Running, Role: LEADER
I20260808 22:25:55.191198 8024 consensus_queue.cc:237] T 907e1a5f97a94e9d8021eb6e009f97db P 7a28325db3974947bc5a325e1d9286c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac0b06ed388e4ead91c8786ff0f52be5" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35059 } } peers { permanent_uuid: "ad6134eec77848e4887b3dec3b1c8bbb" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 33613 } } peers { permanent_uuid: "7a28325db3974947bc5a325e1d9286c6" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 42541 } }
I20260808 22:25:55.194162 7355 catalog_manager.cc:5495] T 907e1a5f97a94e9d8021eb6e009f97db P 7a28325db3974947bc5a325e1d9286c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a28325db3974947bc5a325e1d9286c6 (127.9.176.195). New cstate: current_term: 1 leader_uuid: "7a28325db3974947bc5a325e1d9286c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac0b06ed388e4ead91c8786ff0f52be5" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad6134eec77848e4887b3dec3b1c8bbb" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 33613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a28325db3974947bc5a325e1d9286c6" member_type: VOTER last_known_addr { host: "127.9.176.195" port: 42541 } health_report { overall_health: HEALTHY } } }
W20260808 22:25:55.218401 7989 tablet.cc:2369] T 907e1a5f97a94e9d8021eb6e009f97db P 7a28325db3974947bc5a325e1d9286c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 22:25:55.243870 8009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:55.240731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57616 (local address 127.9.176.195:42541)
0808 22:25:55.240864 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0808 22:25:55.240867 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:25:55.240881 (+ 14us) server_negotiation.cc:407] Connection header received
0808 22:25:55.241024 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:25:55.241026 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:25:55.241098 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:25:55.241184 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:25:55.241367 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:55.241866 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:55.242659 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:55.242859 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:55.242924 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:55.243137 (+ 213us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:25:55.243544 (+ 407us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:25:55.243645 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0808 22:25:55.243692 (+ 47us) server_negotiation.cc:299] Negotiation successful
0808 22:25:55.243739 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 22:25:55.249728 7548 raft_consensus.cc:1261] T 907e1a5f97a94e9d8021eb6e009f97db P ac0b06ed388e4ead91c8786ff0f52be5 [term 1 FOLLOWER]: Refusing update from remote peer 7a28325db3974947bc5a325e1d9286c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:25:55.249774 7745 raft_consensus.cc:1261] T 907e1a5f97a94e9d8021eb6e009f97db P ad6134eec77848e4887b3dec3b1c8bbb [term 1 FOLLOWER]: Refusing update from remote peer 7a28325db3974947bc5a325e1d9286c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:25:55.250387 8029 consensus_queue.cc:1035] T 907e1a5f97a94e9d8021eb6e009f97db P 7a28325db3974947bc5a325e1d9286c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac0b06ed388e4ead91c8786ff0f52be5" member_type: VOTER last_known_addr { host: "127.9.176.193" port: 35059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:25:55.250571 8024 consensus_queue.cc:1035] T 907e1a5f97a94e9d8021eb6e009f97db P 7a28325db3974947bc5a325e1d9286c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad6134eec77848e4887b3dec3b1c8bbb" member_type: VOTER last_known_addr { host: "127.9.176.194" port: 33613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:25:55.257488 8034 mvcc.cc:205] Tried to move back new op lower bound from 7316389704695324672 to 7316389704465764352. Current Snapshot: MvccSnapshot[applied={T|T < 7316389704695324672}]
I20260808 22:25:55.260735 8037 mvcc.cc:205] Tried to move back new op lower bound from 7316389704695324672 to 7316389704465764352. Current Snapshot: MvccSnapshot[applied={T|T < 7316389704695324672}]
I20260808 22:25:55.265767 7355 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41476:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:25:55.265918 7355 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41476:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:25:55.268100 7355 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2a023b17330b415bb0c68603611b5348: Sending DeleteTablet for 3 replicas of tablet 907e1a5f97a94e9d8021eb6e009f97db
I20260808 22:25:55.268807 7893 tablet_service.cc:1514] Processing DeleteTablet for tablet 907e1a5f97a94e9d8021eb6e009f97db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:25:55 UTC) from {username='slave'} at 127.0.0.1:57610
I20260808 22:25:55.268805 7719 tablet_service.cc:1514] Processing DeleteTablet for tablet 907e1a5f97a94e9d8021eb6e009f97db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:25:55 UTC) from {username='slave'} at 127.0.0.1:48454
I20260808 22:25:55.269320 8047 tablet_replica.cc:330] T 907e1a5f97a94e9d8021eb6e009f97db P ad6134eec77848e4887b3dec3b1c8bbb: stopping tablet replica
I20260808 22:25:55.269291 7521 tablet_service.cc:1514] Processing DeleteTablet for tablet 907e1a5f97a94e9d8021eb6e009f97db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:25:55 UTC) from {username='slave'} at 127.0.0.1:59980
I20260808 22:25:55.269578 8048 tablet_replica.cc:330] T 907e1a5f97a94e9d8021eb6e009f97db P 7a28325db3974947bc5a325e1d9286c6: stopping tablet replica
I20260808 22:25:55.269795 8049 tablet_replica.cc:330] T 907e1a5f97a94e9d8021eb6e009f97db P ac0b06ed388e4ead91c8786ff0f52be5: stopping tablet replica
I20260808 22:25:55.270076 8049 raft_consensus.cc:2229] T 907e1a5f97a94e9d8021eb6e009f97db P ac0b06ed388e4ead91c8786ff0f52be5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:25:55.270177 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 7433
I20260808 22:25:55.270320 8048 raft_consensus.cc:2229] T 907e1a5f97a94e9d8021eb6e009f97db P 7a28325db3974947bc5a325e1d9286c6 [term 1 LEADER]: Raft consensus shutting down.
I20260808 22:25:55.270385 8049 raft_consensus.cc:2258] T 907e1a5f97a94e9d8021eb6e009f97db P ac0b06ed388e4ead91c8786ff0f52be5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:25:55.270107 8047 raft_consensus.cc:2229] T 907e1a5f97a94e9d8021eb6e009f97db P ad6134eec77848e4887b3dec3b1c8bbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:25:55.270967 8048 raft_consensus.cc:2258] T 907e1a5f97a94e9d8021eb6e009f97db P 7a28325db3974947bc5a325e1d9286c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:25:55.271322 8047 raft_consensus.cc:2258] T 907e1a5f97a94e9d8021eb6e009f97db P ad6134eec77848e4887b3dec3b1c8bbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:25:55.274860 8048 ts_tablet_manager.cc:1905] T 907e1a5f97a94e9d8021eb6e009f97db P 7a28325db3974947bc5a325e1d9286c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:25:55.278430 8047 ts_tablet_manager.cc:1905] T 907e1a5f97a94e9d8021eb6e009f97db P ad6134eec77848e4887b3dec3b1c8bbb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:25:55.278723 8048 ts_tablet_manager.cc:1918] T 907e1a5f97a94e9d8021eb6e009f97db P 7a28325db3974947bc5a325e1d9286c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 22:25:55.278864 8048 log.cc:1196] T 907e1a5f97a94e9d8021eb6e009f97db P 7a28325db3974947bc5a325e1d9286c6: Deleting WAL directory at /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786227706282457-9923-0/minicluster-data/ts-2/wal/wals/907e1a5f97a94e9d8021eb6e009f97db
I20260808 22:25:55.279217 8048 ts_tablet_manager.cc:1939] T 907e1a5f97a94e9d8021eb6e009f97db P 7a28325db3974947bc5a325e1d9286c6: Deleting consensus metadata
I20260808 22:25:55.280390 7336 catalog_manager.cc:4841] TS 7a28325db3974947bc5a325e1d9286c6 (127.9.176.195:42541): tablet 907e1a5f97a94e9d8021eb6e009f97db (table test-table [id=ccaba8ba6bda4e11ac7f742894f62a9c]) successfully deleted
W20260808 22:25:55.280562 7337 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.176.193:35059 (error 108)
I20260808 22:25:55.280998 8047 ts_tablet_manager.cc:1918] T 907e1a5f97a94e9d8021eb6e009f97db P ad6134eec77848e4887b3dec3b1c8bbb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 22:25:55.281163 8047 log.cc:1196] T 907e1a5f97a94e9d8021eb6e009f97db P ad6134eec77848e4887b3dec3b1c8bbb: Deleting WAL directory at /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786227706282457-9923-0/minicluster-data/ts-1/wal/wals/907e1a5f97a94e9d8021eb6e009f97db
I20260808 22:25:55.281376 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 7630
I20260808 22:25:55.281476 8047 ts_tablet_manager.cc:1939] T 907e1a5f97a94e9d8021eb6e009f97db P ad6134eec77848e4887b3dec3b1c8bbb: Deleting consensus metadata
I20260808 22:25:55.288075 8003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:55.287768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.193:35059 (local address 127.0.0.1:59982)
0808 22:25:55.287909 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:55.287987 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.193:35059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260808 22:25:55.289028 7337 catalog_manager.cc:4584] TS ac0b06ed388e4ead91c8786ff0f52be5 (127.9.176.193:35059): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 907e1a5f97a94e9d8021eb6e009f97db: Network error: Client connection negotiation failed: client connection to 127.9.176.193:35059: connect: Connection refused (error 111)
I20260808 22:25:55.293284 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 7815
I20260808 22:25:55.293551 8003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:55.293283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.194:33613 (local address 127.0.0.1:48460)
0808 22:25:55.293414 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0808 22:25:55.293469 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.194:33613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260808 22:25:55.303491 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 7313
2026-08-08T22:25:55Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1154 ms)
[----------] 6 tests from EncryptionPolicyTest (10447 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786227706282457-9923-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 22:25:55 dist-test-slave-ncp9 krb5kdc[8057](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 22:25:55 dist-test-slave-ncp9 krb5kdc[8057](info): set up 2 sockets
Aug 08 22:25:55 dist-test-slave-ncp9 krb5kdc[8057](info): commencing operation
krb5kdc: starting...
W20260808 22:25:56.432020 9923 mini_kdc.cc:121] Time spent starting KDC: real 1.103s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786227706282457-9923-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786227706282457-9923-0/krb5kdc/test-user.keytab.
Aug 08 22:25:56 dist-test-slave-ncp9 krb5kdc[8057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227956, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T22:25:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:25:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786227706282457-9923-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786227706282457-9923-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 22:25:56.548694 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786227706282457-9923-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786227706282457-9923-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786227706282457-9923-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34187
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786227706282457-9923-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34187
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:25:56.657418 8073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:25:56.657697 8073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:25:56.657756 8073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:25:56.661715 8073 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:25:56.661775 8073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:25:56.661803 8073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:25:56.661823 8073 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:25:56.661851 8073 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:25:56.667166 8073 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786227706282457-9923-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786227706282457-9923-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34187
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786227706282457-9923-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8073
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786227706282457-9923-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:25:56.668210 8073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:25:56.669245 8073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:25:56.677615 8083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:25:56.677626 8082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:25:56.677706 8073 server_base.cc:1034] running on GCE node
W20260808 22:25:56.677626 8085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:25:56.678408 8073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:25:56.679464 8073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:25:56.680624 8073 hybrid_clock.cc:648] HybridClock initialized: now 1786227956680611 us; error 30 us; skew 500 ppm
Aug 08 22:25:56 dist-test-slave-ncp9 krb5kdc[8057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227956, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:25:56.683307 8073 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 22:25:56.684197 8073 webserver.cc:460] Webserver started at http://127.0.0.1:35859/ using document root <none> and password file <none>
I20260808 22:25:56.684718 8073 fs_manager.cc:359] Metadata directory not provided
I20260808 22:25:56.684785 8073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:25:56.685020 8073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:25:56.687055 8073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786227706282457-9923-0/minicluster-data/master-0/data/instance:
uuid: "476548ca195f485f8f61eadccd6094a6"
format_stamp: "Formatted at 2026-08-08 22:25:56 on dist-test-slave-ncp9"
server_key: "f4d339a0707406aa997e04df7051c38a"
server_key_iv: "00f41440694cacabc18f9113ebf634d2"
server_key_version: "encryptionkey@0"
I20260808 22:25:56.687624 8073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786227706282457-9923-0/minicluster-data/master-0/wal/instance:
uuid: "476548ca195f485f8f61eadccd6094a6"
format_stamp: "Formatted at 2026-08-08 22:25:56 on dist-test-slave-ncp9"
server_key: "f4d339a0707406aa997e04df7051c38a"
server_key_iv: "00f41440694cacabc18f9113ebf634d2"
server_key_version: "encryptionkey@0"
I20260808 22:25:56.691267 8073 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260808 22:25:56.693581 8093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:25:56.694756 8073 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 22:25:56.694885 8073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786227706282457-9923-0/minicluster-data/master-0/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786227706282457-9923-0/minicluster-data/master-0/wal
uuid: "476548ca195f485f8f61eadccd6094a6"
format_stamp: "Formatted at 2026-08-08 22:25:56 on dist-test-slave-ncp9"
server_key: "f4d339a0707406aa997e04df7051c38a"
server_key_iv: "00f41440694cacabc18f9113ebf634d2"
server_key_version: "encryptionkey@0"
I20260808 22:25:56.694983 8073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786227706282457-9923-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786227706282457-9923-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786227706282457-9923-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:25:56.722008 8073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:25:56.725059 8073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:25:56.725258 8073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:25:56.732355 8073 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34187
I20260808 22:25:56.732375 8170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34187 every 8 connection(s)
I20260808 22:25:56.733399 8073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
I20260808 22:25:56.734246 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 8073
I20260808 22:25:56.734381 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786227706282457-9923-0/minicluster-data/master-0/wal/instance
I20260808 22:25:56.734648 9923 external_mini_cluster.cc:1428] Setting key def9138a5a5e2c80b3542ef55a7be9a0
I20260808 22:25:56.736588 8172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:25:56.743345 8172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 476548ca195f485f8f61eadccd6094a6: Bootstrap starting.
Aug 08 22:25:56 dist-test-slave-ncp9 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227956, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260808 22:25:56.746006 8172 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 476548ca195f485f8f61eadccd6094a6: Neither blocks nor log segments found. Creating new log.
I20260808 22:25:56.746730 8172 log.cc:824] T 00000000000000000000000000000000 P 476548ca195f485f8f61eadccd6094a6: Log is configured to *not* fsync() on all Append() calls
I20260808 22:25:56.748975 8172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 476548ca195f485f8f61eadccd6094a6: No bootstrap required, opened a new log
I20260808 22:25:56.749980 8175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:56.736088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39859 (local address 127.0.0.1:34187)
0808 22:25:56.736529 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0808 22:25:56.736549 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0808 22:25:56.736595 (+ 46us) server_negotiation.cc:407] Connection header received
0808 22:25:56.737493 (+ 898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:25:56.737526 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:25:56.737969 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:25:56.738329 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:25:56.738967 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:56.740484 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:56.741356 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:56.742039 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:56.742259 (+ 220us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:56.744946 (+ 2687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:25:56.744954 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:25:56.744967 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:25:56.744995 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:25:56.746587 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:25:56.747172 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:25:56.747179 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:25:56.747184 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:25:56.747332 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:25:56.747664 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:25:56.747666 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:25:56.747667 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:25:56.748045 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:25:56.748203 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0808 22:25:56.748508 (+ 305us) server_negotiation.cc:299] Negotiation successful
0808 22:25:56.748721 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":125,"threads_started":1}
I20260808 22:25:56.752919 9923 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 22:25:56.752707 8172 raft_consensus.cc:348] T 00000000000000000000000000000000 P 476548ca195f485f8f61eadccd6094a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "476548ca195f485f8f61eadccd6094a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34187 } }
I20260808 22:25:56.752991 8172 raft_consensus.cc:374] T 00000000000000000000000000000000 P 476548ca195f485f8f61eadccd6094a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:25:56.753057 8172 raft_consensus.cc:729] T 00000000000000000000000000000000 P 476548ca195f485f8f61eadccd6094a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 476548ca195f485f8f61eadccd6094a6, State: Initialized, Role: FOLLOWER
I20260808 22:25:56.753458 8172 consensus_queue.cc:260] T 00000000000000000000000000000000 P 476548ca195f485f8f61eadccd6094a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "476548ca195f485f8f61eadccd6094a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34187 } }
I20260808 22:25:56.753580 8172 raft_consensus.cc:388] T 00000000000000000000000000000000 P 476548ca195f485f8f61eadccd6094a6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:25:56.753640 8172 raft_consensus.cc:482] T 00000000000000000000000000000000 P 476548ca195f485f8f61eadccd6094a6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:25:56.753731 8172 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 476548ca195f485f8f61eadccd6094a6 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:25:56.754673 8172 raft_consensus.cc:504] T 00000000000000000000000000000000 P 476548ca195f485f8f61eadccd6094a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "476548ca195f485f8f61eadccd6094a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34187 } }
I20260808 22:25:56.755023 8172 leader_election.cc:302] T 00000000000000000000000000000000 P 476548ca195f485f8f61eadccd6094a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 476548ca195f485f8f61eadccd6094a6; no voters:
I20260808 22:25:56.755314 8172 leader_election.cc:288] T 00000000000000000000000000000000 P 476548ca195f485f8f61eadccd6094a6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:25:56.755393 8185 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 476548ca195f485f8f61eadccd6094a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:25:56.755592 8185 raft_consensus.cc:686] T 00000000000000000000000000000000 P 476548ca195f485f8f61eadccd6094a6 [term 1 LEADER]: Becoming Leader. State: Replica: 476548ca195f485f8f61eadccd6094a6, State: Running, Role: LEADER
I20260808 22:25:56.755880 8185 consensus_queue.cc:237] T 00000000000000000000000000000000 P 476548ca195f485f8f61eadccd6094a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "476548ca195f485f8f61eadccd6094a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34187 } }
I20260808 22:25:56.756601 8172 sys_catalog.cc:564] T 00000000000000000000000000000000 P 476548ca195f485f8f61eadccd6094a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:25:56.758000 8186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 476548ca195f485f8f61eadccd6094a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "476548ca195f485f8f61eadccd6094a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "476548ca195f485f8f61eadccd6094a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34187 } } }
I20260808 22:25:56.758191 8186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 476548ca195f485f8f61eadccd6094a6 [sys.catalog]: This master's current role is: LEADER
I20260808 22:25:56.758481 8186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 476548ca195f485f8f61eadccd6094a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 476548ca195f485f8f61eadccd6094a6. Latest consensus state: current_term: 1 leader_uuid: "476548ca195f485f8f61eadccd6094a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "476548ca195f485f8f61eadccd6094a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34187 } } }
I20260808 22:25:56.758618 8186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 476548ca195f485f8f61eadccd6094a6 [sys.catalog]: This master's current role is: LEADER
W20260808 22:25:56.761906 8208 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 476548ca195f485f8f61eadccd6094a6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 22:25:56.761984 8208 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 22:25:56.762043 8206 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:25:56.762892 8206 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:25:56.765228 8175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:56.754891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60823 (local address 127.0.0.1:34187)
0808 22:25:56.755016 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0808 22:25:56.755020 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:25:56.755224 (+ 204us) server_negotiation.cc:407] Connection header received
0808 22:25:56.755343 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:25:56.755346 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:25:56.755390 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:25:56.755452 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:25:56.756275 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:56.756959 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:56.760425 (+ 3466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:56.760679 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:56.760765 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:56.762150 (+ 1385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:25:56.762154 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:25:56.762157 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:25:56.762196 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:25:56.763479 (+ 1283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:25:56.763938 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:25:56.763941 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:25:56.763942 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:25:56.764013 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:25:56.764318 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:25:56.764321 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:25:56.764322 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:25:56.764483 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:25:56.764607 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0808 22:25:56.764975 (+ 368us) server_negotiation.cc:299] Negotiation successful
0808 22:25:56.765077 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260808 22:25:56.768497 8206 catalog_manager.cc:1347] Generated new cluster ID: ec8de4edd22240668bdc17157a0fe8c7
I20260808 22:25:56.768555 8206 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:25:56.796145 8206 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:25:56.797500 8206 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:25:56.807112 8206 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 476548ca195f485f8f61eadccd6094a6: Generated new TSK 0
I20260808 22:25:56.807947 8206 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 22:25:56.813628 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 8073
W20260808 22:25:56.827250 8053 connection.cc:535] client connection to 127.0.0.1:34187 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34187): Connection reset by peer (error 104)
2026-08-08T22:25:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1512 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786227706282457-9923-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 22:25:56 dist-test-slave-ncp9 krb5kdc[8215](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 22:25:56 dist-test-slave-ncp9 krb5kdc[8215](info): set up 2 sockets
Aug 08 22:25:56 dist-test-slave-ncp9 krb5kdc[8215](info): commencing operation
krb5kdc: starting...
W20260808 22:25:57.946424 9923 mini_kdc.cc:121] Time spent starting KDC: real 1.105s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786227706282457-9923-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786227706282457-9923-0/krb5kdc/test-user.keytab.
Aug 08 22:25:58 dist-test-slave-ncp9 krb5kdc[8215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T22:25:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:25:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786227706282457-9923-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786227706282457-9923-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 22:25:58.061954 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786227706282457-9923-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786227706282457-9923-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786227706282457-9923-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37643
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786227706282457-9923-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37643
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:25:58.173053 8234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:25:58.173303 8234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:25:58.173346 8234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:25:58.177255 8234 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:25:58.177313 8234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:25:58.177335 8234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:25:58.177359 8234 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:25:58.177382 8234 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:25:58.182627 8234 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786227706282457-9923-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786227706282457-9923-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37643
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786227706282457-9923-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8234
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786227706282457-9923-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:25:58.183796 8234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:25:58.184767 8234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:25:58.193035 8239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:25:58.193048 8247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:25:58.193054 8241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:25:58.193274 8234 server_base.cc:1034] running on GCE node
I20260808 22:25:58.193814 8234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:25:58.194840 8234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:25:58.196005 8234 hybrid_clock.cc:648] HybridClock initialized: now 1786227958195990 us; error 25 us; skew 500 ppm
Aug 08 22:25:58 dist-test-slave-ncp9 krb5kdc[8215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227958, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:25:58.198673 8234 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 22:25:58.199491 8234 webserver.cc:460] Webserver started at http://127.0.0.1:40137/ using document root <none> and password file <none>
I20260808 22:25:58.199997 8234 fs_manager.cc:359] Metadata directory not provided
I20260808 22:25:58.200063 8234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:25:58.200263 8234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:25:58.202270 8234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786227706282457-9923-0/minicluster-data/master-0/data/instance:
uuid: "8d6946732ee94008a66b6865a81c9df4"
format_stamp: "Formatted at 2026-08-08 22:25:58 on dist-test-slave-ncp9"
server_key: "91ada77f7e193b87fe5c013eac6559b3"
server_key_iv: "0aa61d351e52df7be4ab92078cd3f014"
server_key_version: "encryptionkey@0"
I20260808 22:25:58.202816 8234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786227706282457-9923-0/minicluster-data/master-0/wal/instance:
uuid: "8d6946732ee94008a66b6865a81c9df4"
format_stamp: "Formatted at 2026-08-08 22:25:58 on dist-test-slave-ncp9"
server_key: "91ada77f7e193b87fe5c013eac6559b3"
server_key_iv: "0aa61d351e52df7be4ab92078cd3f014"
server_key_version: "encryptionkey@0"
I20260808 22:25:58.206279 8234 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 22:25:58.208463 8256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:25:58.209601 8234 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 22:25:58.209723 8234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786227706282457-9923-0/minicluster-data/master-0/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786227706282457-9923-0/minicluster-data/master-0/wal
uuid: "8d6946732ee94008a66b6865a81c9df4"
format_stamp: "Formatted at 2026-08-08 22:25:58 on dist-test-slave-ncp9"
server_key: "91ada77f7e193b87fe5c013eac6559b3"
server_key_iv: "0aa61d351e52df7be4ab92078cd3f014"
server_key_version: "encryptionkey@0"
I20260808 22:25:58.209813 8234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786227706282457-9923-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786227706282457-9923-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786227706282457-9923-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:25:58.240089 8234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:25:58.243505 8234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:25:58.243701 8234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:25:58.250510 8234 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37643
I20260808 22:25:58.250571 8329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37643 every 8 connection(s)
I20260808 22:25:58.251441 8234 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
I20260808 22:25:58.254201 8331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:25:58.257675 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 8234
I20260808 22:25:58.257822 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786227706282457-9923-0/minicluster-data/master-0/wal/instance
I20260808 22:25:58.258140 9923 external_mini_cluster.cc:1428] Setting key bb878d55543311add4762b14864f7399
I20260808 22:25:58.260154 8331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d6946732ee94008a66b6865a81c9df4: Bootstrap starting.
I20260808 22:25:58.262814 8331 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d6946732ee94008a66b6865a81c9df4: Neither blocks nor log segments found. Creating new log.
I20260808 22:25:58.263664 8331 log.cc:824] T 00000000000000000000000000000000 P 8d6946732ee94008a66b6865a81c9df4: Log is configured to *not* fsync() on all Append() calls
I20260808 22:25:58.265473 8331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d6946732ee94008a66b6865a81c9df4: No bootstrap required, opened a new log
I20260808 22:25:58.267558 8334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:58.259414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38411 (local address 127.0.0.1:37643)
0808 22:25:58.259835 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0808 22:25:58.259855 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0808 22:25:58.259896 (+ 41us) server_negotiation.cc:407] Connection header received
0808 22:25:58.260707 (+ 811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:25:58.260730 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:25:58.261180 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:25:58.261541 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:25:58.261957 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:58.263432 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:58.264308 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:58.264779 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:58.264947 (+ 168us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:58.265827 (+ 880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:25:58.265832 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:25:58.265844 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:25:58.266204 (+ 360us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:25:58.266254 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:25:58.266276 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:25:58.266386 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0808 22:25:58.266577 (+ 191us) server_negotiation.cc:299] Negotiation successful
0808 22:25:58.266705 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":126,"threads_started":1}
I20260808 22:25:58.268389 8331 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d6946732ee94008a66b6865a81c9df4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d6946732ee94008a66b6865a81c9df4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37643 } }
I20260808 22:25:58.268589 8331 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d6946732ee94008a66b6865a81c9df4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:25:58.268633 8331 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d6946732ee94008a66b6865a81c9df4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d6946732ee94008a66b6865a81c9df4, State: Initialized, Role: FOLLOWER
I20260808 22:25:58.269052 8331 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d6946732ee94008a66b6865a81c9df4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d6946732ee94008a66b6865a81c9df4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37643 } }
I20260808 22:25:58.269193 8331 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d6946732ee94008a66b6865a81c9df4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:25:58.269259 8331 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d6946732ee94008a66b6865a81c9df4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:25:58.269358 8331 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d6946732ee94008a66b6865a81c9df4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:25:58.270174 8331 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d6946732ee94008a66b6865a81c9df4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d6946732ee94008a66b6865a81c9df4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37643 } }
I20260808 22:25:58.270263 9923 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 22:25:58.270512 8331 leader_election.cc:302] T 00000000000000000000000000000000 P 8d6946732ee94008a66b6865a81c9df4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d6946732ee94008a66b6865a81c9df4; no voters:
I20260808 22:25:58.270817 8331 leader_election.cc:288] T 00000000000000000000000000000000 P 8d6946732ee94008a66b6865a81c9df4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:25:58.270898 8336 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d6946732ee94008a66b6865a81c9df4 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:25:58.271103 8336 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d6946732ee94008a66b6865a81c9df4 [term 1 LEADER]: Becoming Leader. State: Replica: 8d6946732ee94008a66b6865a81c9df4, State: Running, Role: LEADER
I20260808 22:25:58.271445 8336 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d6946732ee94008a66b6865a81c9df4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d6946732ee94008a66b6865a81c9df4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37643 } }
I20260808 22:25:58.272058 8331 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d6946732ee94008a66b6865a81c9df4 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:25:58.272552 8336 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d6946732ee94008a66b6865a81c9df4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d6946732ee94008a66b6865a81c9df4. Latest consensus state: current_term: 1 leader_uuid: "8d6946732ee94008a66b6865a81c9df4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d6946732ee94008a66b6865a81c9df4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37643 } } }
I20260808 22:25:58.272719 8336 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d6946732ee94008a66b6865a81c9df4 [sys.catalog]: This master's current role is: LEADER
I20260808 22:25:58.273010 8337 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d6946732ee94008a66b6865a81c9df4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d6946732ee94008a66b6865a81c9df4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d6946732ee94008a66b6865a81c9df4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37643 } } }
I20260808 22:25:58.273097 8337 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d6946732ee94008a66b6865a81c9df4 [sys.catalog]: This master's current role is: LEADER
I20260808 22:25:58.273139 8350 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:25:58.276212 8350 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:25:58.278690 8334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:58.272420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38769 (local address 127.0.0.1:37643)
0808 22:25:58.272519 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0808 22:25:58.272523 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:25:58.274246 (+ 1723us) server_negotiation.cc:407] Connection header received
0808 22:25:58.274362 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:25:58.274365 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:25:58.274409 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:25:58.274484 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:25:58.274877 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:58.275625 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:58.276802 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:25:58.277145 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:25:58.277220 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:25:58.278231 (+ 1011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:25:58.278234 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:25:58.278237 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:25:58.278249 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:25:58.278276 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:25:58.278290 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:25:58.278370 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0808 22:25:58.278534 (+ 164us) server_negotiation.cc:299] Negotiation successful
0808 22:25:58.278567 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260808 22:25:58.281440 8350 catalog_manager.cc:1347] Generated new cluster ID: 9f85adf87cd447b0ae4913687b931585
I20260808 22:25:58.281502 8350 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:25:58.306217 8350 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:25:58.307473 8350 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:25:58.321404 8350 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d6946732ee94008a66b6865a81c9df4: Generated new TSK 0
I20260808 22:25:58.322263 8350 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 22:25:58.332679 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 8234
W20260808 22:25:58.342270 8212 connection.cc:535] client connection to 127.0.0.1:37643 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37643): Connection reset by peer (error 104)
2026-08-08T22:25:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1514 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786227706282457-9923-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 22:25:58 dist-test-slave-ncp9 krb5kdc[8376](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 22:25:58 dist-test-slave-ncp9 krb5kdc[8376](info): set up 2 sockets
Aug 08 22:25:58 dist-test-slave-ncp9 krb5kdc[8376](info): commencing operation
krb5kdc: starting...
W20260808 22:25:59.456883 9923 mini_kdc.cc:121] Time spent starting KDC: real 1.101s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786227706282457-9923-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786227706282457-9923-0/krb5kdc/test-user.keytab.
Aug 08 22:25:59 dist-test-slave-ncp9 krb5kdc[8376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T22:25:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:25:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786227706282457-9923-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786227706282457-9923-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 22:25:59.571449 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786227706282457-9923-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786227706282457-9923-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786227706282457-9923-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40911
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786227706282457-9923-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40911
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:25:59.680967 8393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:25:59.681236 8393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:25:59.681295 8393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:25:59.685231 8393 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:25:59.685285 8393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:25:59.685312 8393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:25:59.685333 8393 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:25:59.685361 8393 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:25:59.690637 8393 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786227706282457-9923-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786227706282457-9923-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40911
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786227706282457-9923-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40911
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8393
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786227706282457-9923-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:25:59.691712 8393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:25:59.692734 8393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 22:25:59.701318 8393 server_base.cc:1034] running on GCE node
W20260808 22:25:59.701216 8401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:25:59.701234 8400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:25:59.701244 8404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:25:59.702067 8393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:25:59.703227 8393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:25:59.704403 8393 hybrid_clock.cc:648] HybridClock initialized: now 1786227959704384 us; error 30 us; skew 500 ppm
Aug 08 22:25:59 dist-test-slave-ncp9 krb5kdc[8376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227959, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:25:59.707214 8393 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 22:25:59.708153 8393 webserver.cc:460] Webserver started at http://127.0.0.1:40741/ using document root <none> and password file <none>
I20260808 22:25:59.708706 8393 fs_manager.cc:359] Metadata directory not provided
I20260808 22:25:59.708776 8393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:25:59.709020 8393 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:25:59.711110 8393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786227706282457-9923-0/minicluster-data/master-0/data/instance:
uuid: "115342eb4a754e38b0e429e5b643e43b"
format_stamp: "Formatted at 2026-08-08 22:25:59 on dist-test-slave-ncp9"
server_key: "e0814bf69b6ae75c8ae2619f37cd1608"
server_key_iv: "1df37c355c909dcf8736d58c9dc08e1d"
server_key_version: "encryptionkey@0"
I20260808 22:25:59.711678 8393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786227706282457-9923-0/minicluster-data/master-0/wal/instance:
uuid: "115342eb4a754e38b0e429e5b643e43b"
format_stamp: "Formatted at 2026-08-08 22:25:59 on dist-test-slave-ncp9"
server_key: "e0814bf69b6ae75c8ae2619f37cd1608"
server_key_iv: "1df37c355c909dcf8736d58c9dc08e1d"
server_key_version: "encryptionkey@0"
I20260808 22:25:59.715372 8393 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260808 22:25:59.717777 8411 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:25:59.718990 8393 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 22:25:59.719111 8393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786227706282457-9923-0/minicluster-data/master-0/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786227706282457-9923-0/minicluster-data/master-0/wal
uuid: "115342eb4a754e38b0e429e5b643e43b"
format_stamp: "Formatted at 2026-08-08 22:25:59 on dist-test-slave-ncp9"
server_key: "e0814bf69b6ae75c8ae2619f37cd1608"
server_key_iv: "1df37c355c909dcf8736d58c9dc08e1d"
server_key_version: "encryptionkey@0"
I20260808 22:25:59.719218 8393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786227706282457-9923-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786227706282457-9923-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786227706282457-9923-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:25:59.723757 8393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:25:59.726521 8393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:25:59.726682 8393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:25:59.733327 8393 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40911
I20260808 22:25:59.733372 8489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40911 every 8 connection(s)
I20260808 22:25:59.734226 8393 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
I20260808 22:25:59.736943 8490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:25:59.737078 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 8393
I20260808 22:25:59.737222 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786227706282457-9923-0/minicluster-data/master-0/wal/instance
I20260808 22:25:59.737530 9923 external_mini_cluster.cc:1428] Setting key caab61dcb140cd76a0c84bb51de73c22
I20260808 22:25:59.743044 8495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:59.738921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51263 (local address 127.0.0.1:40911)
0808 22:25:59.739351 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0808 22:25:59.739370 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0808 22:25:59.739413 (+ 43us) server_negotiation.cc:407] Connection header received
0808 22:25:59.740221 (+ 808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:25:59.740247 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:25:59.740601 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:25:59.741002 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:25:59.741687 (+ 685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:25:59.741693 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:25:59.741707 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:25:59.741719 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:25:59.741781 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:25:59.741808 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:25:59.741943 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0808 22:25:59.742011 (+ 68us) server_negotiation.cc:299] Negotiation successful
0808 22:25:59.742155 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":129,"threads_started":1}
I20260808 22:25:59.744181 8490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 115342eb4a754e38b0e429e5b643e43b: Bootstrap starting.
I20260808 22:25:59.745749 9923 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 22:25:59.746474 8490 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 115342eb4a754e38b0e429e5b643e43b: Neither blocks nor log segments found. Creating new log.
I20260808 22:25:59.747197 8490 log.cc:824] T 00000000000000000000000000000000 P 115342eb4a754e38b0e429e5b643e43b: Log is configured to *not* fsync() on all Append() calls
I20260808 22:25:59.749157 8495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:25:59.747713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54955 (local address 127.0.0.1:40911)
0808 22:25:59.747822 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0808 22:25:59.747827 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:25:59.747978 (+ 151us) server_negotiation.cc:407] Connection header received
0808 22:25:59.748094 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:25:59.748096 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:25:59.748121 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:25:59.748182 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:25:59.748813 (+ 631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:25:59.748816 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:25:59.748818 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:25:59.748828 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:25:59.748850 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:25:59.748860 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:25:59.748959 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0808 22:25:59.749039 (+ 80us) server_negotiation.cc:299] Negotiation successful
0808 22:25:59.749068 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260808 22:25:59.749378 8490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 115342eb4a754e38b0e429e5b643e43b: No bootstrap required, opened a new log
I20260808 22:25:59.752717 8490 raft_consensus.cc:348] T 00000000000000000000000000000000 P 115342eb4a754e38b0e429e5b643e43b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "115342eb4a754e38b0e429e5b643e43b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40911 } }
I20260808 22:25:59.752961 8490 raft_consensus.cc:374] T 00000000000000000000000000000000 P 115342eb4a754e38b0e429e5b643e43b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:25:59.753019 8490 raft_consensus.cc:729] T 00000000000000000000000000000000 P 115342eb4a754e38b0e429e5b643e43b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 115342eb4a754e38b0e429e5b643e43b, State: Initialized, Role: FOLLOWER
I20260808 22:25:59.753400 8490 consensus_queue.cc:260] T 00000000000000000000000000000000 P 115342eb4a754e38b0e429e5b643e43b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "115342eb4a754e38b0e429e5b643e43b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40911 } }
I20260808 22:25:59.753523 8490 raft_consensus.cc:388] T 00000000000000000000000000000000 P 115342eb4a754e38b0e429e5b643e43b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:25:59.753587 8490 raft_consensus.cc:482] T 00000000000000000000000000000000 P 115342eb4a754e38b0e429e5b643e43b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:25:59.753688 8490 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 115342eb4a754e38b0e429e5b643e43b [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:25:59.754504 8490 raft_consensus.cc:504] T 00000000000000000000000000000000 P 115342eb4a754e38b0e429e5b643e43b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "115342eb4a754e38b0e429e5b643e43b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40911 } }
I20260808 22:25:59.754822 8490 leader_election.cc:302] T 00000000000000000000000000000000 P 115342eb4a754e38b0e429e5b643e43b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 115342eb4a754e38b0e429e5b643e43b; no voters:
I20260808 22:25:59.755097 8490 leader_election.cc:288] T 00000000000000000000000000000000 P 115342eb4a754e38b0e429e5b643e43b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:25:59.755185 8506 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 115342eb4a754e38b0e429e5b643e43b [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:25:59.755383 8506 raft_consensus.cc:686] T 00000000000000000000000000000000 P 115342eb4a754e38b0e429e5b643e43b [term 1 LEADER]: Becoming Leader. State: Replica: 115342eb4a754e38b0e429e5b643e43b, State: Running, Role: LEADER
I20260808 22:25:59.755723 8506 consensus_queue.cc:237] T 00000000000000000000000000000000 P 115342eb4a754e38b0e429e5b643e43b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "115342eb4a754e38b0e429e5b643e43b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40911 } }
I20260808 22:25:59.756278 8490 sys_catalog.cc:564] T 00000000000000000000000000000000 P 115342eb4a754e38b0e429e5b643e43b [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:25:59.757426 8511 sys_catalog.cc:455] T 00000000000000000000000000000000 P 115342eb4a754e38b0e429e5b643e43b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "115342eb4a754e38b0e429e5b643e43b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "115342eb4a754e38b0e429e5b643e43b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40911 } } }
I20260808 22:25:59.757551 8511 sys_catalog.cc:458] T 00000000000000000000000000000000 P 115342eb4a754e38b0e429e5b643e43b [sys.catalog]: This master's current role is: LEADER
I20260808 22:25:59.757745 8507 sys_catalog.cc:455] T 00000000000000000000000000000000 P 115342eb4a754e38b0e429e5b643e43b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 115342eb4a754e38b0e429e5b643e43b. Latest consensus state: current_term: 1 leader_uuid: "115342eb4a754e38b0e429e5b643e43b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "115342eb4a754e38b0e429e5b643e43b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40911 } } }
I20260808 22:25:59.757895 8507 sys_catalog.cc:458] T 00000000000000000000000000000000 P 115342eb4a754e38b0e429e5b643e43b [sys.catalog]: This master's current role is: LEADER
I20260808 22:25:59.757970 8519 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:25:59.760706 8519 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:25:59.766144 8519 catalog_manager.cc:1347] Generated new cluster ID: 13ba89fb49774c7282c102a702885a77
I20260808 22:25:59.766194 8519 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:25:59.786177 8519 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:25:59.786466 8519 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:25:59.796794 8519 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 115342eb4a754e38b0e429e5b643e43b: Generated new TSK 0
I20260808 22:25:59.797567 8519 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 22:25:59.807950 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 8393
2026-08-08T22:25:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1477 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786227706282457-9923-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 22:25:59 dist-test-slave-ncp9 krb5kdc[8533](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 22:25:59 dist-test-slave-ncp9 krb5kdc[8533](info): set up 2 sockets
Aug 08 22:25:59 dist-test-slave-ncp9 krb5kdc[8533](info): commencing operation
krb5kdc: starting...
W20260808 22:26:00.934894 9923 mini_kdc.cc:121] Time spent starting KDC: real 1.101s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786227706282457-9923-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786227706282457-9923-0/krb5kdc/test-user.keytab.
Aug 08 22:26:01 dist-test-slave-ncp9 krb5kdc[8533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227961, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T22:26:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:26:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786227706282457-9923-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786227706282457-9923-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 22:26:01.051551 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786227706282457-9923-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786227706282457-9923-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786227706282457-9923-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34731
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786227706282457-9923-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34731
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:26:01.160744 8553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:26:01.161063 8553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:26:01.161110 8553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:26:01.165014 8553 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:26:01.165079 8553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:26:01.165098 8553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:26:01.165113 8553 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:26:01.165127 8553 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:26:01.170329 8553 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786227706282457-9923-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786227706282457-9923-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34731
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786227706282457-9923-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34731
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8553
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786227706282457-9923-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:26:01.171314 8553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:26:01.172261 8553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:26:01.180738 8564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:26:01.180768 8559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:26:01.180768 8560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:26:01.180812 8553 server_base.cc:1034] running on GCE node
I20260808 22:26:01.181524 8553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:26:01.182624 8553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:26:01.183815 8553 hybrid_clock.cc:648] HybridClock initialized: now 1786227961183798 us; error 32 us; skew 500 ppm
Aug 08 22:26:01 dist-test-slave-ncp9 krb5kdc[8533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:26:01.190526 8553 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 22:26:01.191439 8553 webserver.cc:460] Webserver started at http://127.0.0.1:45759/ using document root <none> and password file <none>
I20260808 22:26:01.191967 8553 fs_manager.cc:359] Metadata directory not provided
I20260808 22:26:01.192034 8553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:26:01.192234 8553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:26:01.194295 8553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786227706282457-9923-0/minicluster-data/master-0/data/instance:
uuid: "742fe51c2cb74d8684cf7bc7c2e8f402"
format_stamp: "Formatted at 2026-08-08 22:26:01 on dist-test-slave-ncp9"
server_key: "55b5a0c93713f62e5a455fe3cfc33f82"
server_key_iv: "8495021b786fc2f5cfad9a0a9cd8b1f5"
server_key_version: "encryptionkey@0"
I20260808 22:26:01.194818 8553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786227706282457-9923-0/minicluster-data/master-0/wal/instance:
uuid: "742fe51c2cb74d8684cf7bc7c2e8f402"
format_stamp: "Formatted at 2026-08-08 22:26:01 on dist-test-slave-ncp9"
server_key: "55b5a0c93713f62e5a455fe3cfc33f82"
server_key_iv: "8495021b786fc2f5cfad9a0a9cd8b1f5"
server_key_version: "encryptionkey@0"
I20260808 22:26:01.198603 8553 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 22:26:01.201037 8574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:26:01.202270 8553 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 22:26:01.202406 8553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786227706282457-9923-0/minicluster-data/master-0/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786227706282457-9923-0/minicluster-data/master-0/wal
uuid: "742fe51c2cb74d8684cf7bc7c2e8f402"
format_stamp: "Formatted at 2026-08-08 22:26:01 on dist-test-slave-ncp9"
server_key: "55b5a0c93713f62e5a455fe3cfc33f82"
server_key_iv: "8495021b786fc2f5cfad9a0a9cd8b1f5"
server_key_version: "encryptionkey@0"
I20260808 22:26:01.202505 8553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786227706282457-9923-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786227706282457-9923-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786227706282457-9923-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:26:01.232717 8553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:26:01.235644 8553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:26:01.235834 8553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:26:01.243014 8553 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34731
I20260808 22:26:01.243069 8637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34731 every 8 connection(s)
I20260808 22:26:01.243986 8553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
I20260808 22:26:01.246793 8639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:26:01.247560 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 8553
I20260808 22:26:01.247680 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786227706282457-9923-0/minicluster-data/master-0/wal/instance
I20260808 22:26:01.247962 9923 external_mini_cluster.cc:1428] Setting key 7f9f8ae31d39dc04706f75c9e5e915a8
I20260808 22:26:01.253363 8639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 742fe51c2cb74d8684cf7bc7c2e8f402: Bootstrap starting.
I20260808 22:26:01.255815 8639 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 742fe51c2cb74d8684cf7bc7c2e8f402: Neither blocks nor log segments found. Creating new log.
I20260808 22:26:01.256541 8639 log.cc:824] T 00000000000000000000000000000000 P 742fe51c2cb74d8684cf7bc7c2e8f402: Log is configured to *not* fsync() on all Append() calls
Aug 08 22:26:01 dist-test-slave-ncp9 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227961, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260808 22:26:01.258524 8639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 742fe51c2cb74d8684cf7bc7c2e8f402: No bootstrap required, opened a new log
I20260808 22:26:01.262748 8642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:01.249401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57597 (local address 127.0.0.1:34731)
0808 22:26:01.249839 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0808 22:26:01.249858 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0808 22:26:01.249903 (+ 45us) server_negotiation.cc:407] Connection header received
0808 22:26:01.250713 (+ 810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:26:01.250737 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:26:01.251155 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:26:01.251531 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:26:01.252215 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:01.253718 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:01.254670 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:01.255327 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:01.255484 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:26:01.257878 (+ 2394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:26:01.257884 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:26:01.257899 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:26:01.257928 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:26:01.259763 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:26:01.260156 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:26:01.260163 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:26:01.260168 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:26:01.260301 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:26:01.260580 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:26:01.260583 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:26:01.260585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:26:01.261120 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:26:01.261300 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0808 22:26:01.261500 (+ 200us) server_negotiation.cc:299] Negotiation successful
0808 22:26:01.261785 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":110,"threads_started":1}
I20260808 22:26:01.262832 8639 raft_consensus.cc:348] T 00000000000000000000000000000000 P 742fe51c2cb74d8684cf7bc7c2e8f402 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "742fe51c2cb74d8684cf7bc7c2e8f402" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34731 } }
I20260808 22:26:01.263092 8639 raft_consensus.cc:374] T 00000000000000000000000000000000 P 742fe51c2cb74d8684cf7bc7c2e8f402 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:26:01.263159 8639 raft_consensus.cc:729] T 00000000000000000000000000000000 P 742fe51c2cb74d8684cf7bc7c2e8f402 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 742fe51c2cb74d8684cf7bc7c2e8f402, State: Initialized, Role: FOLLOWER
I20260808 22:26:01.263579 8639 consensus_queue.cc:260] T 00000000000000000000000000000000 P 742fe51c2cb74d8684cf7bc7c2e8f402 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "742fe51c2cb74d8684cf7bc7c2e8f402" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34731 } }
I20260808 22:26:01.263710 8639 raft_consensus.cc:388] T 00000000000000000000000000000000 P 742fe51c2cb74d8684cf7bc7c2e8f402 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:26:01.263775 8639 raft_consensus.cc:482] T 00000000000000000000000000000000 P 742fe51c2cb74d8684cf7bc7c2e8f402 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:26:01.263866 8639 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 742fe51c2cb74d8684cf7bc7c2e8f402 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:26:01.264703 8639 raft_consensus.cc:504] T 00000000000000000000000000000000 P 742fe51c2cb74d8684cf7bc7c2e8f402 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "742fe51c2cb74d8684cf7bc7c2e8f402" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34731 } }
I20260808 22:26:01.265058 8639 leader_election.cc:302] T 00000000000000000000000000000000 P 742fe51c2cb74d8684cf7bc7c2e8f402 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 742fe51c2cb74d8684cf7bc7c2e8f402; no voters:
I20260808 22:26:01.265353 8639 leader_election.cc:288] T 00000000000000000000000000000000 P 742fe51c2cb74d8684cf7bc7c2e8f402 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:26:01.265432 8644 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 742fe51c2cb74d8684cf7bc7c2e8f402 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:26:01.265484 9923 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 22:26:01.265656 8644 raft_consensus.cc:686] T 00000000000000000000000000000000 P 742fe51c2cb74d8684cf7bc7c2e8f402 [term 1 LEADER]: Becoming Leader. State: Replica: 742fe51c2cb74d8684cf7bc7c2e8f402, State: Running, Role: LEADER
I20260808 22:26:01.265946 8644 consensus_queue.cc:237] T 00000000000000000000000000000000 P 742fe51c2cb74d8684cf7bc7c2e8f402 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "742fe51c2cb74d8684cf7bc7c2e8f402" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34731 } }
I20260808 22:26:01.266599 8639 sys_catalog.cc:564] T 00000000000000000000000000000000 P 742fe51c2cb74d8684cf7bc7c2e8f402 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:26:01.267081 8644 sys_catalog.cc:455] T 00000000000000000000000000000000 P 742fe51c2cb74d8684cf7bc7c2e8f402 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 742fe51c2cb74d8684cf7bc7c2e8f402. Latest consensus state: current_term: 1 leader_uuid: "742fe51c2cb74d8684cf7bc7c2e8f402" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "742fe51c2cb74d8684cf7bc7c2e8f402" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34731 } } }
I20260808 22:26:01.267180 8644 sys_catalog.cc:458] T 00000000000000000000000000000000 P 742fe51c2cb74d8684cf7bc7c2e8f402 [sys.catalog]: This master's current role is: LEADER
I20260808 22:26:01.267448 8645 sys_catalog.cc:455] T 00000000000000000000000000000000 P 742fe51c2cb74d8684cf7bc7c2e8f402 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "742fe51c2cb74d8684cf7bc7c2e8f402" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "742fe51c2cb74d8684cf7bc7c2e8f402" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34731 } } }
I20260808 22:26:01.267549 8645 sys_catalog.cc:458] T 00000000000000000000000000000000 P 742fe51c2cb74d8684cf7bc7c2e8f402 [sys.catalog]: This master's current role is: LEADER
W20260808 22:26:01.271903 8673 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 742fe51c2cb74d8684cf7bc7c2e8f402: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 22:26:01.271981 8673 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 22:26:01.272119 8660 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:26:01.272982 8660 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:26:01.275852 8642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:01.267716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37383 (local address 127.0.0.1:34731)
0808 22:26:01.267817 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0808 22:26:01.267820 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:26:01.268012 (+ 192us) server_negotiation.cc:407] Connection header received
0808 22:26:01.268161 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:26:01.268164 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:26:01.268211 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:26:01.268289 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:26:01.269205 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:01.269833 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:01.270616 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:01.270846 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:01.270929 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:26:01.272113 (+ 1184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:26:01.272117 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:26:01.272120 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:26:01.272141 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:26:01.273732 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:26:01.274211 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:26:01.274214 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:26:01.274216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:26:01.274302 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:26:01.274583 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:26:01.274586 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:26:01.274587 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:26:01.274792 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:26:01.274909 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0808 22:26:01.275591 (+ 682us) server_negotiation.cc:299] Negotiation successful
0808 22:26:01.275706 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260808 22:26:01.277938 8660 catalog_manager.cc:1347] Generated new cluster ID: 24cec694738e42efb2872624435442d6
I20260808 22:26:01.277998 8660 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:26:01.291481 8660 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:26:01.292531 8660 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:26:01.301095 8660 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 742fe51c2cb74d8684cf7bc7c2e8f402: Generated new TSK 0
I20260808 22:26:01.301854 8660 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 22:26:01.311676 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 8553
2026-08-08T22:26:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1504 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786227706282457-9923-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 22:26:01 dist-test-slave-ncp9 krb5kdc[8682](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 22:26:01 dist-test-slave-ncp9 krb5kdc[8682](info): set up 2 sockets
Aug 08 22:26:01 dist-test-slave-ncp9 krb5kdc[8682](info): commencing operation
krb5kdc: starting...
W20260808 22:26:02.436697 9923 mini_kdc.cc:121] Time spent starting KDC: real 1.098s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786227706282457-9923-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786227706282457-9923-0/krb5kdc/test-user.keytab.
Aug 08 22:26:02 dist-test-slave-ncp9 krb5kdc[8682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227962, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T22:26:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:26:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786227706282457-9923-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786227706282457-9923-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 22:26:02.549417 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786227706282457-9923-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786227706282457-9923-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786227706282457-9923-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38501
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786227706282457-9923-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38501
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:26:02.656154 8698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:26:02.656406 8698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:26:02.656459 8698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:26:02.660625 8698 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:26:02.660699 8698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:26:02.660719 8698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:26:02.660751 8698 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:26:02.660779 8698 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:26:02.665943 8698 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786227706282457-9923-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786227706282457-9923-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38501
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786227706282457-9923-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8698
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786227706282457-9923-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:26:02.666904 8698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:26:02.667860 8698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:26:02.675895 8705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:26:02.675895 8707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:26:02.675987 8698 server_base.cc:1034] running on GCE node
W20260808 22:26:02.675907 8709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:26:02.676671 8698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:26:02.677773 8698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:26:02.678936 8698 hybrid_clock.cc:648] HybridClock initialized: now 1786227962678919 us; error 28 us; skew 500 ppm
Aug 08 22:26:02 dist-test-slave-ncp9 krb5kdc[8682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227962, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:26:02.681628 8698 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 22:26:02.682504 8698 webserver.cc:460] Webserver started at http://127.0.0.1:35525/ using document root <none> and password file <none>
I20260808 22:26:02.683017 8698 fs_manager.cc:359] Metadata directory not provided
I20260808 22:26:02.683084 8698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:26:02.683269 8698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:26:02.685285 8698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786227706282457-9923-0/minicluster-data/master-0/data/instance:
uuid: "8acfd23c01ba46d6a8ca139ba69e1436"
format_stamp: "Formatted at 2026-08-08 22:26:02 on dist-test-slave-ncp9"
server_key: "3a1938714df814c7227d9dcac1b880c1"
server_key_iv: "b9706931d87af219cac456a3aacb1914"
server_key_version: "encryptionkey@0"
I20260808 22:26:02.685830 8698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786227706282457-9923-0/minicluster-data/master-0/wal/instance:
uuid: "8acfd23c01ba46d6a8ca139ba69e1436"
format_stamp: "Formatted at 2026-08-08 22:26:02 on dist-test-slave-ncp9"
server_key: "3a1938714df814c7227d9dcac1b880c1"
server_key_iv: "b9706931d87af219cac456a3aacb1914"
server_key_version: "encryptionkey@0"
I20260808 22:26:02.689383 8698 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260808 22:26:02.691646 8721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:26:02.692755 8698 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 22:26:02.692870 8698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786227706282457-9923-0/minicluster-data/master-0/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786227706282457-9923-0/minicluster-data/master-0/wal
uuid: "8acfd23c01ba46d6a8ca139ba69e1436"
format_stamp: "Formatted at 2026-08-08 22:26:02 on dist-test-slave-ncp9"
server_key: "3a1938714df814c7227d9dcac1b880c1"
server_key_iv: "b9706931d87af219cac456a3aacb1914"
server_key_version: "encryptionkey@0"
I20260808 22:26:02.693011 8698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786227706282457-9923-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786227706282457-9923-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786227706282457-9923-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:26:02.704401 8698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:26:02.707311 8698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:26:02.707518 8698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:26:02.716223 8698 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38501
I20260808 22:26:02.716234 8788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38501 every 8 connection(s)
I20260808 22:26:02.717229 8698 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
I20260808 22:26:02.719933 8792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:26:02.724376 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 8698
I20260808 22:26:02.724557 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786227706282457-9923-0/minicluster-data/master-0/wal/instance
I20260808 22:26:02.724803 9923 external_mini_cluster.cc:1428] Setting key 1033125b67d23eed0857b7e0eb92aaeb
I20260808 22:26:02.725564 8792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8acfd23c01ba46d6a8ca139ba69e1436: Bootstrap starting.
I20260808 22:26:02.728130 8792 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8acfd23c01ba46d6a8ca139ba69e1436: Neither blocks nor log segments found. Creating new log.
I20260808 22:26:02.729040 8792 log.cc:824] T 00000000000000000000000000000000 P 8acfd23c01ba46d6a8ca139ba69e1436: Log is configured to *not* fsync() on all Append() calls
I20260808 22:26:02.731019 8792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8acfd23c01ba46d6a8ca139ba69e1436: No bootstrap required, opened a new log
I20260808 22:26:02.734794 8797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:02.726249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56699 (local address 127.0.0.1:38501)
0808 22:26:02.726687 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0808 22:26:02.726711 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0808 22:26:02.726756 (+ 45us) server_negotiation.cc:407] Connection header received
0808 22:26:02.727565 (+ 809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:26:02.727588 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:26:02.728014 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:26:02.728395 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:26:02.728818 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:02.730159 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:02.731077 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:02.731772 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:02.731938 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:26:02.732726 (+ 788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:26:02.732732 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:26:02.732748 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:26:02.733268 (+ 520us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:26:02.733335 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:26:02.733363 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:26:02.733453 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0808 22:26:02.733626 (+ 173us) server_negotiation.cc:299] Negotiation successful
0808 22:26:02.733848 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":112,"threads_started":1}
I20260808 22:26:02.735383 8792 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8acfd23c01ba46d6a8ca139ba69e1436 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8acfd23c01ba46d6a8ca139ba69e1436" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38501 } }
I20260808 22:26:02.735605 8792 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8acfd23c01ba46d6a8ca139ba69e1436 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:26:02.735661 8792 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8acfd23c01ba46d6a8ca139ba69e1436 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8acfd23c01ba46d6a8ca139ba69e1436, State: Initialized, Role: FOLLOWER
I20260808 22:26:02.736110 8792 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8acfd23c01ba46d6a8ca139ba69e1436 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8acfd23c01ba46d6a8ca139ba69e1436" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38501 } }
I20260808 22:26:02.736261 8792 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8acfd23c01ba46d6a8ca139ba69e1436 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:26:02.736327 8792 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8acfd23c01ba46d6a8ca139ba69e1436 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:26:02.736423 8792 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8acfd23c01ba46d6a8ca139ba69e1436 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:26:02.737421 8792 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8acfd23c01ba46d6a8ca139ba69e1436 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8acfd23c01ba46d6a8ca139ba69e1436" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38501 } }
I20260808 22:26:02.737661 9923 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 22:26:02.737776 8792 leader_election.cc:302] T 00000000000000000000000000000000 P 8acfd23c01ba46d6a8ca139ba69e1436 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8acfd23c01ba46d6a8ca139ba69e1436; no voters:
I20260808 22:26:02.738072 8792 leader_election.cc:288] T 00000000000000000000000000000000 P 8acfd23c01ba46d6a8ca139ba69e1436 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:26:02.738161 8801 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8acfd23c01ba46d6a8ca139ba69e1436 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:26:02.738386 8801 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8acfd23c01ba46d6a8ca139ba69e1436 [term 1 LEADER]: Becoming Leader. State: Replica: 8acfd23c01ba46d6a8ca139ba69e1436, State: Running, Role: LEADER
I20260808 22:26:02.738771 8801 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8acfd23c01ba46d6a8ca139ba69e1436 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8acfd23c01ba46d6a8ca139ba69e1436" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38501 } }
I20260808 22:26:02.739326 8792 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8acfd23c01ba46d6a8ca139ba69e1436 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:26:02.740567 8804 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8acfd23c01ba46d6a8ca139ba69e1436 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8acfd23c01ba46d6a8ca139ba69e1436. Latest consensus state: current_term: 1 leader_uuid: "8acfd23c01ba46d6a8ca139ba69e1436" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8acfd23c01ba46d6a8ca139ba69e1436" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38501 } } }
I20260808 22:26:02.740670 8804 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8acfd23c01ba46d6a8ca139ba69e1436 [sys.catalog]: This master's current role is: LEADER
I20260808 22:26:02.741070 8817 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:26:02.741183 8802 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8acfd23c01ba46d6a8ca139ba69e1436 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8acfd23c01ba46d6a8ca139ba69e1436" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8acfd23c01ba46d6a8ca139ba69e1436" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38501 } } }
I20260808 22:26:02.741274 8802 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8acfd23c01ba46d6a8ca139ba69e1436 [sys.catalog]: This master's current role is: LEADER
I20260808 22:26:02.744201 8817 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:26:02.745187 8797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:02.740973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33075 (local address 127.0.0.1:38501)
0808 22:26:02.741071 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0808 22:26:02.741075 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:26:02.741100 (+ 25us) server_negotiation.cc:407] Connection header received
0808 22:26:02.741138 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:26:02.741142 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:26:02.741183 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:26:02.741261 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:26:02.741637 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:02.742263 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:02.743325 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:02.743662 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:02.743763 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:26:02.744569 (+ 806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:26:02.744572 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:26:02.744575 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:26:02.744587 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:26:02.744613 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:26:02.744626 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:26:02.744696 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0808 22:26:02.744999 (+ 303us) server_negotiation.cc:299] Negotiation successful
0808 22:26:02.745050 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260808 22:26:02.749585 8817 catalog_manager.cc:1347] Generated new cluster ID: f27d21b4ca5e4e47bb90d1ec6d066efd
I20260808 22:26:02.749646 8817 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:26:02.753916 8817 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:26:02.755040 8817 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:26:02.762950 8817 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8acfd23c01ba46d6a8ca139ba69e1436: Generated new TSK 0
I20260808 22:26:02.763617 8817 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 22:26:02.767630 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 8698
2026-08-08T22:26:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1456 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786227706282457-9923-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 22:26:02 dist-test-slave-ncp9 krb5kdc[8836](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 22:26:02 dist-test-slave-ncp9 krb5kdc[8836](info): set up 2 sockets
Aug 08 22:26:02 dist-test-slave-ncp9 krb5kdc[8836](info): commencing operation
krb5kdc: starting...
W20260808 22:26:03.882165 9923 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786227706282457-9923-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786227706282457-9923-0/krb5kdc/test-user.keytab.
Aug 08 22:26:03 dist-test-slave-ncp9 krb5kdc[8836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227963, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T22:26:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:26:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786227706282457-9923-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786227706282457-9923-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 22:26:03.996860 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786227706282457-9923-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786227706282457-9923-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786227706282457-9923-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36751
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786227706282457-9923-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36751
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:26:04.103510 8854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:26:04.103742 8854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:26:04.103781 8854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:26:04.107604 8854 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:26:04.107652 8854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:26:04.107669 8854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:26:04.107684 8854 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:26:04.107697 8854 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:26:04.112728 8854 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786227706282457-9923-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786227706282457-9923-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36751
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786227706282457-9923-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8854
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786227706282457-9923-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:26:04.113708 8854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:26:04.114637 8854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 22:26:04.122817 8854 server_base.cc:1034] running on GCE node
W20260808 22:26:04.122762 8865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:26:04.122747 8868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:26:04.122839 8861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:26:04.123579 8854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:26:04.124629 8854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:26:04.125814 8854 hybrid_clock.cc:648] HybridClock initialized: now 1786227964125778 us; error 50 us; skew 500 ppm
Aug 08 22:26:04 dist-test-slave-ncp9 krb5kdc[8836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227964, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:26:04.128446 8854 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 22:26:04.129331 8854 webserver.cc:460] Webserver started at http://127.0.0.1:33315/ using document root <none> and password file <none>
I20260808 22:26:04.129822 8854 fs_manager.cc:359] Metadata directory not provided
I20260808 22:26:04.129879 8854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:26:04.130043 8854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:26:04.131991 8854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786227706282457-9923-0/minicluster-data/master-0/data/instance:
uuid: "b5598da565364ebcb4a0d563970be154"
format_stamp: "Formatted at 2026-08-08 22:26:04 on dist-test-slave-ncp9"
server_key: "af53ca14454217950377ee2d9f68eb6b"
server_key_iv: "854bd5cf9fadd0cafbdf1f1bb6f524ab"
server_key_version: "encryptionkey@0"
I20260808 22:26:04.132498 8854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786227706282457-9923-0/minicluster-data/master-0/wal/instance:
uuid: "b5598da565364ebcb4a0d563970be154"
format_stamp: "Formatted at 2026-08-08 22:26:04 on dist-test-slave-ncp9"
server_key: "af53ca14454217950377ee2d9f68eb6b"
server_key_iv: "854bd5cf9fadd0cafbdf1f1bb6f524ab"
server_key_version: "encryptionkey@0"
I20260808 22:26:04.135931 8854 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260808 22:26:04.138363 8876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:26:04.139629 8854 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 22:26:04.139732 8854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786227706282457-9923-0/minicluster-data/master-0/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786227706282457-9923-0/minicluster-data/master-0/wal
uuid: "b5598da565364ebcb4a0d563970be154"
format_stamp: "Formatted at 2026-08-08 22:26:04 on dist-test-slave-ncp9"
server_key: "af53ca14454217950377ee2d9f68eb6b"
server_key_iv: "854bd5cf9fadd0cafbdf1f1bb6f524ab"
server_key_version: "encryptionkey@0"
I20260808 22:26:04.139816 8854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786227706282457-9923-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786227706282457-9923-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786227706282457-9923-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:26:04.144115 8854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:26:04.146880 8854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:26:04.147019 8854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:26:04.153903 8854 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36751
I20260808 22:26:04.153919 8947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36751 every 8 connection(s)
I20260808 22:26:04.154759 8854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
I20260808 22:26:04.157373 8948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:26:04.161962 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 8854
I20260808 22:26:04.162091 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786227706282457-9923-0/minicluster-data/master-0/wal/instance
I20260808 22:26:04.162348 9923 external_mini_cluster.cc:1428] Setting key 8579e03e6f683dbf295dc407b542c141
I20260808 22:26:04.162600 8948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5598da565364ebcb4a0d563970be154: Bootstrap starting.
I20260808 22:26:04.164944 8948 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5598da565364ebcb4a0d563970be154: Neither blocks nor log segments found. Creating new log.
I20260808 22:26:04.165714 8948 log.cc:824] T 00000000000000000000000000000000 P b5598da565364ebcb4a0d563970be154: Log is configured to *not* fsync() on all Append() calls
I20260808 22:26:04.167218 8951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:04.163553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50913 (local address 127.0.0.1:36751)
0808 22:26:04.163984 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0808 22:26:04.163998 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 22:26:04.164032 (+ 34us) server_negotiation.cc:407] Connection header received
0808 22:26:04.164628 (+ 596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:26:04.164643 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:26:04.164956 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:26:04.165289 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:26:04.166090 (+ 801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:26:04.166096 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:26:04.166109 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:26:04.166119 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:26:04.166173 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:26:04.166192 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:26:04.166282 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0808 22:26:04.166410 (+ 128us) server_negotiation.cc:299] Negotiation successful
0808 22:26:04.166563 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":120,"threads_started":1}
I20260808 22:26:04.167553 8948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5598da565364ebcb4a0d563970be154: No bootstrap required, opened a new log
I20260808 22:26:04.170138 9923 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 22:26:04.171545 8948 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5598da565364ebcb4a0d563970be154 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5598da565364ebcb4a0d563970be154" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36751 } }
I20260808 22:26:04.171756 8948 raft_consensus.cc:374] T 00000000000000000000000000000000 P b5598da565364ebcb4a0d563970be154 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:26:04.171809 8948 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5598da565364ebcb4a0d563970be154 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5598da565364ebcb4a0d563970be154, State: Initialized, Role: FOLLOWER
I20260808 22:26:04.172252 8948 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5598da565364ebcb4a0d563970be154 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5598da565364ebcb4a0d563970be154" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36751 } }
I20260808 22:26:04.172420 8948 raft_consensus.cc:388] T 00000000000000000000000000000000 P b5598da565364ebcb4a0d563970be154 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:26:04.172492 8948 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5598da565364ebcb4a0d563970be154 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:26:04.172603 8948 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5598da565364ebcb4a0d563970be154 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:26:04.173600 8951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:04.172051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56035 (local address 127.0.0.1:36751)
0808 22:26:04.172150 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0808 22:26:04.172153 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:26:04.172322 (+ 169us) server_negotiation.cc:407] Connection header received
0808 22:26:04.172474 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:26:04.172478 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:26:04.172513 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:26:04.172586 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:26:04.173272 (+ 686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:26:04.173276 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:26:04.173278 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:26:04.173289 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:26:04.173316 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:26:04.173328 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:26:04.173408 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0808 22:26:04.173457 (+ 49us) server_negotiation.cc:299] Negotiation successful
0808 22:26:04.173492 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260808 22:26:04.173702 8948 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5598da565364ebcb4a0d563970be154 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5598da565364ebcb4a0d563970be154" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36751 } }
I20260808 22:26:04.174038 8948 leader_election.cc:302] T 00000000000000000000000000000000 P b5598da565364ebcb4a0d563970be154 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5598da565364ebcb4a0d563970be154; no voters:
I20260808 22:26:04.174312 8948 leader_election.cc:288] T 00000000000000000000000000000000 P b5598da565364ebcb4a0d563970be154 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:26:04.174389 8959 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5598da565364ebcb4a0d563970be154 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:26:04.174561 8959 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5598da565364ebcb4a0d563970be154 [term 1 LEADER]: Becoming Leader. State: Replica: b5598da565364ebcb4a0d563970be154, State: Running, Role: LEADER
I20260808 22:26:04.174896 8959 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5598da565364ebcb4a0d563970be154 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5598da565364ebcb4a0d563970be154" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36751 } }
I20260808 22:26:04.175534 8948 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5598da565364ebcb4a0d563970be154 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:26:04.176436 8961 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5598da565364ebcb4a0d563970be154 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5598da565364ebcb4a0d563970be154. Latest consensus state: current_term: 1 leader_uuid: "b5598da565364ebcb4a0d563970be154" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5598da565364ebcb4a0d563970be154" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36751 } } }
I20260808 22:26:04.176785 8961 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5598da565364ebcb4a0d563970be154 [sys.catalog]: This master's current role is: LEADER
I20260808 22:26:04.176688 8960 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5598da565364ebcb4a0d563970be154 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b5598da565364ebcb4a0d563970be154" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5598da565364ebcb4a0d563970be154" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36751 } } }
I20260808 22:26:04.177134 8960 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5598da565364ebcb4a0d563970be154 [sys.catalog]: This master's current role is: LEADER
I20260808 22:26:04.177227 8974 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:26:04.179760 8974 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:26:04.184621 8974 catalog_manager.cc:1347] Generated new cluster ID: b34b92a286a947b3939082eb4ce5de5f
I20260808 22:26:04.184684 8974 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:26:04.203454 8974 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:26:04.203779 8974 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:26:04.216955 8974 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b5598da565364ebcb4a0d563970be154: Generated new TSK 0
I20260808 22:26:04.217728 8974 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 22:26:04.221159 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 8854
2026-08-08T22:26:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1451 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786227706282457-9923-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 22:26:04 dist-test-slave-ncp9 krb5kdc[8995](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 22:26:04 dist-test-slave-ncp9 krb5kdc[8995](info): set up 2 sockets
Aug 08 22:26:04 dist-test-slave-ncp9 krb5kdc[8995](info): commencing operation
krb5kdc: starting...
W20260808 22:26:05.355788 9923 mini_kdc.cc:121] Time spent starting KDC: real 1.109s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786227706282457-9923-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786227706282457-9923-0/krb5kdc/test-user.keytab.
Aug 08 22:26:05 dist-test-slave-ncp9 krb5kdc[8995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227965, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T22:26:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:26:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786227706282457-9923-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786227706282457-9923-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 22:26:05.475697 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786227706282457-9923-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786227706282457-9923-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786227706282457-9923-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41671
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786227706282457-9923-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41671
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:26:05.583603 9012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:26:05.583848 9012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:26:05.583904 9012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:26:05.587795 9012 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:26:05.587850 9012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:26:05.587877 9012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:26:05.587898 9012 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:26:05.587921 9012 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:26:05.592944 9012 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786227706282457-9923-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786227706282457-9923-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41671
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786227706282457-9923-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9012
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786227706282457-9923-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:26:05.593925 9012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:26:05.594882 9012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:26:05.602958 9018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:26:05.603016 9019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:26:05.602969 9021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:26:05.603072 9012 server_base.cc:1034] running on GCE node
I20260808 22:26:05.603796 9012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:26:05.604820 9012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:26:05.605978 9012 hybrid_clock.cc:648] HybridClock initialized: now 1786227965605962 us; error 26 us; skew 500 ppm
Aug 08 22:26:05 dist-test-slave-ncp9 krb5kdc[8995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227965, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:26:05.608515 9012 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 22:26:05.609368 9012 webserver.cc:460] Webserver started at http://127.0.0.1:38471/ using document root <none> and password file <none>
I20260808 22:26:05.609879 9012 fs_manager.cc:359] Metadata directory not provided
I20260808 22:26:05.609943 9012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:26:05.610128 9012 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:26:05.612082 9012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786227706282457-9923-0/minicluster-data/master-0/data/instance:
uuid: "6c4873c392ea4c0985e645369d74fdac"
format_stamp: "Formatted at 2026-08-08 22:26:05 on dist-test-slave-ncp9"
server_key: "151b1245e7fb8315fa8f026f20b7d1ae"
server_key_iv: "74758e4b73314231f0a64012cc01b9c8"
server_key_version: "encryptionkey@0"
I20260808 22:26:05.612596 9012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786227706282457-9923-0/minicluster-data/master-0/wal/instance:
uuid: "6c4873c392ea4c0985e645369d74fdac"
format_stamp: "Formatted at 2026-08-08 22:26:05 on dist-test-slave-ncp9"
server_key: "151b1245e7fb8315fa8f026f20b7d1ae"
server_key_iv: "74758e4b73314231f0a64012cc01b9c8"
server_key_version: "encryptionkey@0"
I20260808 22:26:05.616109 9012 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260808 22:26:05.618408 9029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:26:05.619510 9012 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 22:26:05.619621 9012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786227706282457-9923-0/minicluster-data/master-0/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786227706282457-9923-0/minicluster-data/master-0/wal
uuid: "6c4873c392ea4c0985e645369d74fdac"
format_stamp: "Formatted at 2026-08-08 22:26:05 on dist-test-slave-ncp9"
server_key: "151b1245e7fb8315fa8f026f20b7d1ae"
server_key_iv: "74758e4b73314231f0a64012cc01b9c8"
server_key_version: "encryptionkey@0"
I20260808 22:26:05.619724 9012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786227706282457-9923-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786227706282457-9923-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786227706282457-9923-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:26:05.648734 9012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:26:05.651986 9012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:26:05.652176 9012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:26:05.659111 9012 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41671
I20260808 22:26:05.659140 9097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41671 every 8 connection(s)
I20260808 22:26:05.660069 9012 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
I20260808 22:26:05.661208 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 9012
I20260808 22:26:05.661350 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786227706282457-9923-0/minicluster-data/master-0/wal/instance
I20260808 22:26:05.661696 9923 external_mini_cluster.cc:1428] Setting key 3f31386fcdd1a93fd0a528450a9dfb84
I20260808 22:26:05.663295 9098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:26:05.668764 9098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c4873c392ea4c0985e645369d74fdac: Bootstrap starting.
Aug 08 22:26:05 dist-test-slave-ncp9 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227965, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260808 22:26:05.671284 9098 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c4873c392ea4c0985e645369d74fdac: Neither blocks nor log segments found. Creating new log.
I20260808 22:26:05.671972 9098 log.cc:824] T 00000000000000000000000000000000 P 6c4873c392ea4c0985e645369d74fdac: Log is configured to *not* fsync() on all Append() calls
I20260808 22:26:05.674003 9098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c4873c392ea4c0985e645369d74fdac: No bootstrap required, opened a new log
I20260808 22:26:05.675676 9103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:05.663256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.213:35981 (local address 127.0.0.1:41671)
0808 22:26:05.663705 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0808 22:26:05.663720 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 22:26:05.663757 (+ 37us) server_negotiation.cc:407] Connection header received
0808 22:26:05.664408 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:26:05.664424 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:26:05.664762 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:26:05.665113 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:26:05.665912 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:05.666917 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:05.667765 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:05.668245 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:05.668389 (+ 144us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:26:05.671144 (+ 2755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:26:05.671149 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:26:05.671169 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:26:05.671195 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:26:05.672741 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:26:05.673291 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:26:05.673296 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:26:05.673300 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:26:05.673404 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:26:05.673725 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:26:05.673727 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:26:05.673728 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:26:05.674080 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:26:05.674267 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0808 22:26:05.674542 (+ 275us) server_negotiation.cc:299] Negotiation successful
0808 22:26:05.674742 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":155,"threads_started":1}
I20260808 22:26:05.677187 9098 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c4873c392ea4c0985e645369d74fdac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c4873c392ea4c0985e645369d74fdac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41671 } }
I20260808 22:26:05.677438 9098 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c4873c392ea4c0985e645369d74fdac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:26:05.677529 9098 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c4873c392ea4c0985e645369d74fdac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c4873c392ea4c0985e645369d74fdac, State: Initialized, Role: FOLLOWER
I20260808 22:26:05.677958 9098 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c4873c392ea4c0985e645369d74fdac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c4873c392ea4c0985e645369d74fdac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41671 } }
I20260808 22:26:05.678098 9098 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c4873c392ea4c0985e645369d74fdac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:26:05.678164 9098 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c4873c392ea4c0985e645369d74fdac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:26:05.678253 9098 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c4873c392ea4c0985e645369d74fdac [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:26:05.678551 9923 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 22:26:05.679265 9098 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c4873c392ea4c0985e645369d74fdac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c4873c392ea4c0985e645369d74fdac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41671 } }
I20260808 22:26:05.679620 9098 leader_election.cc:302] T 00000000000000000000000000000000 P 6c4873c392ea4c0985e645369d74fdac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c4873c392ea4c0985e645369d74fdac; no voters:
I20260808 22:26:05.679957 9098 leader_election.cc:288] T 00000000000000000000000000000000 P 6c4873c392ea4c0985e645369d74fdac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:26:05.680022 9109 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c4873c392ea4c0985e645369d74fdac [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:26:05.680269 9109 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c4873c392ea4c0985e645369d74fdac [term 1 LEADER]: Becoming Leader. State: Replica: 6c4873c392ea4c0985e645369d74fdac, State: Running, Role: LEADER
I20260808 22:26:05.680609 9109 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c4873c392ea4c0985e645369d74fdac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c4873c392ea4c0985e645369d74fdac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41671 } }
I20260808 22:26:05.681320 9098 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c4873c392ea4c0985e645369d74fdac [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:26:05.682303 9112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c4873c392ea4c0985e645369d74fdac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c4873c392ea4c0985e645369d74fdac. Latest consensus state: current_term: 1 leader_uuid: "6c4873c392ea4c0985e645369d74fdac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c4873c392ea4c0985e645369d74fdac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41671 } } }
I20260808 22:26:05.682423 9112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c4873c392ea4c0985e645369d74fdac [sys.catalog]: This master's current role is: LEADER
I20260808 22:26:05.682787 9113 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c4873c392ea4c0985e645369d74fdac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c4873c392ea4c0985e645369d74fdac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c4873c392ea4c0985e645369d74fdac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41671 } } }
I20260808 22:26:05.682931 9113 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c4873c392ea4c0985e645369d74fdac [sys.catalog]: This master's current role is: LEADER
W20260808 22:26:05.686462 9135 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6c4873c392ea4c0985e645369d74fdac: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 22:26:05.686574 9135 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 22:26:05.686674 9123 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:26:05.687465 9123 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:26:05.688848 9103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:05.680741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.213:50703 (local address 127.0.0.1:41671)
0808 22:26:05.680837 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0808 22:26:05.680841 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:26:05.681104 (+ 263us) server_negotiation.cc:407] Connection header received
0808 22:26:05.681216 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:26:05.681219 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:26:05.681263 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:26:05.681340 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:26:05.682107 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:05.682878 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:05.683981 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:05.684207 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:05.684296 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:26:05.685600 (+ 1304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:26:05.685604 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:26:05.685606 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:26:05.685629 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:26:05.686969 (+ 1340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:26:05.687517 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:26:05.687521 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:26:05.687523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:26:05.687606 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:26:05.687977 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:26:05.687980 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:26:05.687982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:26:05.688183 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:26:05.688324 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0808 22:26:05.688612 (+ 288us) server_negotiation.cc:299] Negotiation successful
0808 22:26:05.688708 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260808 22:26:05.693234 9123 catalog_manager.cc:1347] Generated new cluster ID: 42d3f81b14d343b9b5c017c84ba63935
I20260808 22:26:05.693300 9123 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:26:05.702105 9123 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:26:05.703327 9123 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:26:05.710703 9123 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c4873c392ea4c0985e645369d74fdac: Generated new TSK 0
I20260808 22:26:05.711476 9123 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 22:26:05.723985 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 9012
W20260808 22:26:05.740614 8991 connection.cc:535] client connection to 127.0.0.1:41671 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41671): Connection reset by peer (error 104)
2026-08-08T22:26:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1508 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786227706282457-9923-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 22:26:05 dist-test-slave-ncp9 krb5kdc[9141](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 22:26:05 dist-test-slave-ncp9 krb5kdc[9141](info): set up 2 sockets
Aug 08 22:26:05 dist-test-slave-ncp9 krb5kdc[9141](info): commencing operation
krb5kdc: starting...
W20260808 22:26:06.844713 9923 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786227706282457-9923-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786227706282457-9923-0/krb5kdc/test-user.keytab.
Aug 08 22:26:06 dist-test-slave-ncp9 krb5kdc[9141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T22:26:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:26:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786227706282457-9923-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786227706282457-9923-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 22:26:06.960806 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786227706282457-9923-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786227706282457-9923-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786227706282457-9923-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43503
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786227706282457-9923-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43503
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:26:07.069026 9157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:26:07.069262 9157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:26:07.069299 9157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:26:07.073143 9157 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:26:07.073191 9157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:26:07.073210 9157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:26:07.073225 9157 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:26:07.073237 9157 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:26:07.078279 9157 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786227706282457-9923-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786227706282457-9923-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43503
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786227706282457-9923-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43503
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9157
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786227706282457-9923-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:26:07.079226 9157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:26:07.080281 9157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:26:07.088585 9164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:26:07.088616 9165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:26:07.088588 9168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:26:07.088773 9157 server_base.cc:1034] running on GCE node
I20260808 22:26:07.089422 9157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:26:07.090416 9157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:26:07.091576 9157 hybrid_clock.cc:648] HybridClock initialized: now 1786227967091560 us; error 25 us; skew 500 ppm
Aug 08 22:26:07 dist-test-slave-ncp9 krb5kdc[9141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227967, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:26:07.098289 9157 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 22:26:07.099260 9157 webserver.cc:460] Webserver started at http://127.0.0.1:38895/ using document root <none> and password file <none>
I20260808 22:26:07.099792 9157 fs_manager.cc:359] Metadata directory not provided
I20260808 22:26:07.099859 9157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:26:07.100054 9157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:26:07.102114 9157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786227706282457-9923-0/minicluster-data/master-0/data/instance:
uuid: "d579781339c94647b8622999ecdf863f"
format_stamp: "Formatted at 2026-08-08 22:26:07 on dist-test-slave-ncp9"
server_key: "6d7c41322958ee6f82f3e607a8c5a8b2"
server_key_iv: "01fc5f2f4459bac9fdc5705753e40718"
server_key_version: "encryptionkey@0"
I20260808 22:26:07.102665 9157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786227706282457-9923-0/minicluster-data/master-0/wal/instance:
uuid: "d579781339c94647b8622999ecdf863f"
format_stamp: "Formatted at 2026-08-08 22:26:07 on dist-test-slave-ncp9"
server_key: "6d7c41322958ee6f82f3e607a8c5a8b2"
server_key_iv: "01fc5f2f4459bac9fdc5705753e40718"
server_key_version: "encryptionkey@0"
I20260808 22:26:07.106312 9157 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 22:26:07.108700 9175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:26:07.109890 9157 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260808 22:26:07.110008 9157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786227706282457-9923-0/minicluster-data/master-0/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786227706282457-9923-0/minicluster-data/master-0/wal
uuid: "d579781339c94647b8622999ecdf863f"
format_stamp: "Formatted at 2026-08-08 22:26:07 on dist-test-slave-ncp9"
server_key: "6d7c41322958ee6f82f3e607a8c5a8b2"
server_key_iv: "01fc5f2f4459bac9fdc5705753e40718"
server_key_version: "encryptionkey@0"
I20260808 22:26:07.110103 9157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786227706282457-9923-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786227706282457-9923-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786227706282457-9923-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:26:07.122226 9157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:26:07.125154 9157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:26:07.125329 9157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:26:07.131922 9157 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43503
I20260808 22:26:07.131946 9244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43503 every 8 connection(s)
I20260808 22:26:07.132808 9157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
I20260808 22:26:07.135466 9246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:26:07.136348 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 9157
I20260808 22:26:07.136487 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786227706282457-9923-0/minicluster-data/master-0/wal/instance
I20260808 22:26:07.136816 9923 external_mini_cluster.cc:1428] Setting key 47566b180372c445a8d9cc2d82ef8298
I20260808 22:26:07.141482 9246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d579781339c94647b8622999ecdf863f: Bootstrap starting.
I20260808 22:26:07.144038 9246 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d579781339c94647b8622999ecdf863f: Neither blocks nor log segments found. Creating new log.
I20260808 22:26:07.144760 9246 log.cc:824] T 00000000000000000000000000000000 P d579781339c94647b8622999ecdf863f: Log is configured to *not* fsync() on all Append() calls
I20260808 22:26:07.145853 9251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:07.138218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.213:54301 (local address 127.0.0.1:43503)
0808 22:26:07.138645 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0808 22:26:07.138659 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 22:26:07.138694 (+ 35us) server_negotiation.cc:407] Connection header received
0808 22:26:07.139330 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:26:07.139347 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:26:07.139711 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:26:07.140114 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:26:07.140508 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:07.141528 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:07.142778 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:07.143230 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:07.143339 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:26:07.144227 (+ 888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:26:07.144232 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:26:07.144244 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:26:07.144613 (+ 369us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:26:07.144665 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:26:07.144687 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:26:07.144782 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0808 22:26:07.145024 (+ 242us) server_negotiation.cc:299] Negotiation successful
0808 22:26:07.145152 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":131,"threads_started":1}
I20260808 22:26:07.146752 9246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d579781339c94647b8622999ecdf863f: No bootstrap required, opened a new log
I20260808 22:26:07.148586 9923 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 22:26:07.150331 9246 raft_consensus.cc:348] T 00000000000000000000000000000000 P d579781339c94647b8622999ecdf863f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d579781339c94647b8622999ecdf863f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43503 } }
I20260808 22:26:07.150594 9246 raft_consensus.cc:374] T 00000000000000000000000000000000 P d579781339c94647b8622999ecdf863f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:26:07.150660 9246 raft_consensus.cc:729] T 00000000000000000000000000000000 P d579781339c94647b8622999ecdf863f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d579781339c94647b8622999ecdf863f, State: Initialized, Role: FOLLOWER
I20260808 22:26:07.151095 9246 consensus_queue.cc:260] T 00000000000000000000000000000000 P d579781339c94647b8622999ecdf863f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d579781339c94647b8622999ecdf863f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43503 } }
I20260808 22:26:07.151260 9246 raft_consensus.cc:388] T 00000000000000000000000000000000 P d579781339c94647b8622999ecdf863f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:26:07.151335 9246 raft_consensus.cc:482] T 00000000000000000000000000000000 P d579781339c94647b8622999ecdf863f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:26:07.151444 9246 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d579781339c94647b8622999ecdf863f [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:26:07.152383 9246 raft_consensus.cc:504] T 00000000000000000000000000000000 P d579781339c94647b8622999ecdf863f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d579781339c94647b8622999ecdf863f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43503 } }
I20260808 22:26:07.152743 9246 leader_election.cc:302] T 00000000000000000000000000000000 P d579781339c94647b8622999ecdf863f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d579781339c94647b8622999ecdf863f; no voters:
I20260808 22:26:07.153105 9246 leader_election.cc:288] T 00000000000000000000000000000000 P d579781339c94647b8622999ecdf863f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:26:07.153188 9260 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d579781339c94647b8622999ecdf863f [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:26:07.153362 9260 raft_consensus.cc:686] T 00000000000000000000000000000000 P d579781339c94647b8622999ecdf863f [term 1 LEADER]: Becoming Leader. State: Replica: d579781339c94647b8622999ecdf863f, State: Running, Role: LEADER
I20260808 22:26:07.153647 9260 consensus_queue.cc:237] T 00000000000000000000000000000000 P d579781339c94647b8622999ecdf863f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d579781339c94647b8622999ecdf863f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43503 } }
I20260808 22:26:07.154335 9246 sys_catalog.cc:564] T 00000000000000000000000000000000 P d579781339c94647b8622999ecdf863f [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:26:07.154843 9251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:07.150480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.213:58901 (local address 127.0.0.1:43503)
0808 22:26:07.150584 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0808 22:26:07.150588 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:26:07.150755 (+ 167us) server_negotiation.cc:407] Connection header received
0808 22:26:07.150925 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:26:07.150928 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:26:07.150976 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:26:07.151048 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:26:07.151362 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:07.151918 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:07.153015 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:07.153306 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:07.153795 (+ 489us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:26:07.154256 (+ 461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:26:07.154259 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:26:07.154262 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:26:07.154274 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:26:07.154299 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:26:07.154309 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:26:07.154379 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0808 22:26:07.154682 (+ 303us) server_negotiation.cc:299] Negotiation successful
0808 22:26:07.154720 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260808 22:26:07.154824 9260 sys_catalog.cc:455] T 00000000000000000000000000000000 P d579781339c94647b8622999ecdf863f [sys.catalog]: SysCatalogTable state changed. Reason: New leader d579781339c94647b8622999ecdf863f. Latest consensus state: current_term: 1 leader_uuid: "d579781339c94647b8622999ecdf863f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d579781339c94647b8622999ecdf863f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43503 } } }
I20260808 22:26:07.155223 9260 sys_catalog.cc:458] T 00000000000000000000000000000000 P d579781339c94647b8622999ecdf863f [sys.catalog]: This master's current role is: LEADER
I20260808 22:26:07.155546 9261 sys_catalog.cc:455] T 00000000000000000000000000000000 P d579781339c94647b8622999ecdf863f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d579781339c94647b8622999ecdf863f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d579781339c94647b8622999ecdf863f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43503 } } }
I20260808 22:26:07.155697 9261 sys_catalog.cc:458] T 00000000000000000000000000000000 P d579781339c94647b8622999ecdf863f [sys.catalog]: This master's current role is: LEADER
I20260808 22:26:07.155818 9271 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:26:07.158757 9271 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:26:07.163605 9271 catalog_manager.cc:1347] Generated new cluster ID: dbd565d419f34b248702275b104eac1e
I20260808 22:26:07.163671 9271 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:26:07.184661 9271 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:26:07.185834 9271 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:26:07.191267 9271 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d579781339c94647b8622999ecdf863f: Generated new TSK 0
I20260808 22:26:07.191901 9271 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 22:26:07.199369 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 9157
W20260808 22:26:07.209803 9138 connection.cc:535] client connection to 127.0.0.1:43503 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43503): Connection reset by peer (error 104)
2026-08-08T22:26:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1469 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786227706282457-9923-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 22:26:07 dist-test-slave-ncp9 krb5kdc[9289](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 22:26:07 dist-test-slave-ncp9 krb5kdc[9289](info): set up 2 sockets
Aug 08 22:26:07 dist-test-slave-ncp9 krb5kdc[9289](info): commencing operation
krb5kdc: starting...
W20260808 22:26:08.324085 9923 mini_kdc.cc:121] Time spent starting KDC: real 1.100s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786227706282457-9923-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786227706282457-9923-0/krb5kdc/test-user.keytab.
Aug 08 22:26:08 dist-test-slave-ncp9 krb5kdc[9289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T22:26:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:26:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786227706282457-9923-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786227706282457-9923-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 22:26:08.437942 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786227706282457-9923-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786227706282457-9923-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786227706282457-9923-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41845
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786227706282457-9923-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41845
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:26:08.546284 9305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:26:08.546520 9305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:26:08.546558 9305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:26:08.550427 9305 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:26:08.550478 9305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:26:08.550495 9305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:26:08.550509 9305 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:26:08.550523 9305 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:26:08.555557 9305 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786227706282457-9923-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786227706282457-9923-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41845
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786227706282457-9923-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9305
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786227706282457-9923-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:26:08.556493 9305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:26:08.557480 9305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:26:08.565444 9315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:26:08.565429 9313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:26:08.565560 9305 server_base.cc:1034] running on GCE node
W20260808 22:26:08.565552 9318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:26:08.566181 9305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:26:08.567176 9305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:26:08.568331 9305 hybrid_clock.cc:648] HybridClock initialized: now 1786227968568315 us; error 28 us; skew 500 ppm
Aug 08 22:26:08 dist-test-slave-ncp9 krb5kdc[9289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227968, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:26:08.571014 9305 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 22:26:08.571862 9305 webserver.cc:460] Webserver started at http://127.0.0.1:44859/ using document root <none> and password file <none>
I20260808 22:26:08.572368 9305 fs_manager.cc:359] Metadata directory not provided
I20260808 22:26:08.572427 9305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:26:08.572618 9305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:26:08.574651 9305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786227706282457-9923-0/minicluster-data/master-0/data/instance:
uuid: "3af56e3a39f7465aabc7c5c9367dbdf3"
format_stamp: "Formatted at 2026-08-08 22:26:08 on dist-test-slave-ncp9"
server_key: "b9a64c1e90c1c2b5eb713c8752cdfd2c"
server_key_iv: "c54e8f2a1887f1a0c422b61038af0aca"
server_key_version: "encryptionkey@0"
I20260808 22:26:08.575189 9305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786227706282457-9923-0/minicluster-data/master-0/wal/instance:
uuid: "3af56e3a39f7465aabc7c5c9367dbdf3"
format_stamp: "Formatted at 2026-08-08 22:26:08 on dist-test-slave-ncp9"
server_key: "b9a64c1e90c1c2b5eb713c8752cdfd2c"
server_key_iv: "c54e8f2a1887f1a0c422b61038af0aca"
server_key_version: "encryptionkey@0"
I20260808 22:26:08.578742 9305 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 22:26:08.580977 9325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:26:08.582087 9305 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 22:26:08.582217 9305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786227706282457-9923-0/minicluster-data/master-0/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786227706282457-9923-0/minicluster-data/master-0/wal
uuid: "3af56e3a39f7465aabc7c5c9367dbdf3"
format_stamp: "Formatted at 2026-08-08 22:26:08 on dist-test-slave-ncp9"
server_key: "b9a64c1e90c1c2b5eb713c8752cdfd2c"
server_key_iv: "c54e8f2a1887f1a0c422b61038af0aca"
server_key_version: "encryptionkey@0"
I20260808 22:26:08.582309 9305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786227706282457-9923-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786227706282457-9923-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786227706282457-9923-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:26:08.586547 9305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:26:08.589309 9305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:26:08.589464 9305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:26:08.596122 9305 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41845
I20260808 22:26:08.596138 9393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41845 every 8 connection(s)
I20260808 22:26:08.597078 9305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
I20260808 22:26:08.599663 9395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:26:08.603482 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 9305
I20260808 22:26:08.603626 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786227706282457-9923-0/minicluster-data/master-0/wal/instance
I20260808 22:26:08.603909 9923 external_mini_cluster.cc:1428] Setting key 938c6634baebe89fc15b16ad78e7d706
I20260808 22:26:08.605363 9395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3af56e3a39f7465aabc7c5c9367dbdf3: Bootstrap starting.
I20260808 22:26:08.607939 9395 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3af56e3a39f7465aabc7c5c9367dbdf3: Neither blocks nor log segments found. Creating new log.
I20260808 22:26:08.608841 9395 log.cc:824] T 00000000000000000000000000000000 P 3af56e3a39f7465aabc7c5c9367dbdf3: Log is configured to *not* fsync() on all Append() calls
I20260808 22:26:08.609375 9398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:08.605228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.213:41587 (local address 127.0.0.1:41845)
0808 22:26:08.605638 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0808 22:26:08.605661 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0808 22:26:08.605701 (+ 40us) server_negotiation.cc:407] Connection header received
0808 22:26:08.606496 (+ 795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:26:08.606520 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:26:08.606863 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:26:08.607212 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:26:08.607867 (+ 655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:26:08.607872 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:26:08.607888 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:26:08.607901 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:26:08.607975 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:26:08.608008 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:26:08.608158 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0808 22:26:08.608299 (+ 141us) server_negotiation.cc:299] Negotiation successful
0808 22:26:08.608458 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":108,"threads_started":1}
I20260808 22:26:08.610841 9395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3af56e3a39f7465aabc7c5c9367dbdf3: No bootstrap required, opened a new log
I20260808 22:26:08.612039 9923 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 22:26:08.614552 9395 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3af56e3a39f7465aabc7c5c9367dbdf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3af56e3a39f7465aabc7c5c9367dbdf3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41845 } }
I20260808 22:26:08.614781 9395 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3af56e3a39f7465aabc7c5c9367dbdf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:26:08.614837 9395 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3af56e3a39f7465aabc7c5c9367dbdf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3af56e3a39f7465aabc7c5c9367dbdf3, State: Initialized, Role: FOLLOWER
I20260808 22:26:08.615224 9395 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3af56e3a39f7465aabc7c5c9367dbdf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3af56e3a39f7465aabc7c5c9367dbdf3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41845 } }
I20260808 22:26:08.615350 9395 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3af56e3a39f7465aabc7c5c9367dbdf3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:26:08.615447 9395 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3af56e3a39f7465aabc7c5c9367dbdf3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:26:08.615475 9398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:08.613951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.213:56309 (local address 127.0.0.1:41845)
0808 22:26:08.614071 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0808 22:26:08.614075 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:26:08.614258 (+ 183us) server_negotiation.cc:407] Connection header received
0808 22:26:08.614360 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:26:08.614363 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:26:08.614393 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:26:08.614452 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:26:08.615127 (+ 675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:26:08.615131 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:26:08.615133 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:26:08.615144 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:26:08.615169 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:26:08.615180 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:26:08.615247 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0808 22:26:08.615320 (+ 73us) server_negotiation.cc:299] Negotiation successful
0808 22:26:08.615357 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 22:26:08.615557 9395 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3af56e3a39f7465aabc7c5c9367dbdf3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:26:08.616539 9395 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3af56e3a39f7465aabc7c5c9367dbdf3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3af56e3a39f7465aabc7c5c9367dbdf3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41845 } }
I20260808 22:26:08.616844 9395 leader_election.cc:302] T 00000000000000000000000000000000 P 3af56e3a39f7465aabc7c5c9367dbdf3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3af56e3a39f7465aabc7c5c9367dbdf3; no voters:
I20260808 22:26:08.617147 9395 leader_election.cc:288] T 00000000000000000000000000000000 P 3af56e3a39f7465aabc7c5c9367dbdf3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:26:08.617235 9405 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3af56e3a39f7465aabc7c5c9367dbdf3 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:26:08.617431 9405 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3af56e3a39f7465aabc7c5c9367dbdf3 [term 1 LEADER]: Becoming Leader. State: Replica: 3af56e3a39f7465aabc7c5c9367dbdf3, State: Running, Role: LEADER
I20260808 22:26:08.617756 9405 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3af56e3a39f7465aabc7c5c9367dbdf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3af56e3a39f7465aabc7c5c9367dbdf3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41845 } }
I20260808 22:26:08.618242 9395 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3af56e3a39f7465aabc7c5c9367dbdf3 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:26:08.619194 9410 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3af56e3a39f7465aabc7c5c9367dbdf3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3af56e3a39f7465aabc7c5c9367dbdf3. Latest consensus state: current_term: 1 leader_uuid: "3af56e3a39f7465aabc7c5c9367dbdf3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3af56e3a39f7465aabc7c5c9367dbdf3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41845 } } }
I20260808 22:26:08.619315 9410 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3af56e3a39f7465aabc7c5c9367dbdf3 [sys.catalog]: This master's current role is: LEADER
I20260808 22:26:08.619535 9406 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3af56e3a39f7465aabc7c5c9367dbdf3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3af56e3a39f7465aabc7c5c9367dbdf3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3af56e3a39f7465aabc7c5c9367dbdf3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41845 } } }
I20260808 22:26:08.619625 9406 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3af56e3a39f7465aabc7c5c9367dbdf3 [sys.catalog]: This master's current role is: LEADER
I20260808 22:26:08.619885 9422 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:26:08.622807 9422 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:26:08.627914 9422 catalog_manager.cc:1347] Generated new cluster ID: 2dcaf7aeee7247f98c4edf2eda8820dc
I20260808 22:26:08.627967 9422 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:26:08.639161 9422 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:26:08.639433 9422 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:26:08.651997 9422 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3af56e3a39f7465aabc7c5c9367dbdf3: Generated new TSK 0
I20260808 22:26:08.652649 9422 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 22:26:08.656615 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 9305
2026-08-08T22:26:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1457 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786227706282457-9923-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 22:26:08 dist-test-slave-ncp9 krb5kdc[9441](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 22:26:08 dist-test-slave-ncp9 krb5kdc[9441](info): set up 2 sockets
Aug 08 22:26:08 dist-test-slave-ncp9 krb5kdc[9441](info): commencing operation
krb5kdc: starting...
W20260808 22:26:09.775164 9923 mini_kdc.cc:121] Time spent starting KDC: real 1.094s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786227706282457-9923-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786227706282457-9923-0/krb5kdc/test-user.keytab.
Aug 08 22:26:09 dist-test-slave-ncp9 krb5kdc[9441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227969, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T22:26:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:26:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786227706282457-9923-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786227706282457-9923-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 22:26:09.891758 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786227706282457-9923-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786227706282457-9923-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786227706282457-9923-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35637
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786227706282457-9923-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35637
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:26:09.998713 9462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:26:09.998939 9462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:26:09.998986 9462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:26:10.002835 9462 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:26:10.002888 9462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:26:10.002907 9462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:26:10.002920 9462 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:26:10.002933 9462 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:26:10.007951 9462 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786227706282457-9923-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786227706282457-9923-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35637
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786227706282457-9923-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35637
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9462
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786227706282457-9923-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:26:10.008877 9462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:26:10.009840 9462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:26:10.018059 9467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:26:10.018059 9469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:26:10.018095 9462 server_base.cc:1034] running on GCE node
W20260808 22:26:10.018059 9471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:26:10.018814 9462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:26:10.019814 9462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:26:10.020954 9462 hybrid_clock.cc:648] HybridClock initialized: now 1786227970020931 us; error 38 us; skew 500 ppm
Aug 08 22:26:10 dist-test-slave-ncp9 krb5kdc[9441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227970, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:26:10.023587 9462 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 22:26:10.024446 9462 webserver.cc:460] Webserver started at http://127.0.0.1:42765/ using document root <none> and password file <none>
I20260808 22:26:10.025005 9462 fs_manager.cc:359] Metadata directory not provided
I20260808 22:26:10.025076 9462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:26:10.025274 9462 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:26:10.027263 9462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786227706282457-9923-0/minicluster-data/master-0/data/instance:
uuid: "9abb97d53c6d45bba360075b5fe344ad"
format_stamp: "Formatted at 2026-08-08 22:26:10 on dist-test-slave-ncp9"
server_key: "ad905cf36c979b9683c39a2dacf2a2ca"
server_key_iv: "fbf0a6e26a3ef1d9c505dc10c88026fc"
server_key_version: "encryptionkey@0"
I20260808 22:26:10.027796 9462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786227706282457-9923-0/minicluster-data/master-0/wal/instance:
uuid: "9abb97d53c6d45bba360075b5fe344ad"
format_stamp: "Formatted at 2026-08-08 22:26:10 on dist-test-slave-ncp9"
server_key: "ad905cf36c979b9683c39a2dacf2a2ca"
server_key_iv: "fbf0a6e26a3ef1d9c505dc10c88026fc"
server_key_version: "encryptionkey@0"
I20260808 22:26:10.031317 9462 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260808 22:26:10.033586 9482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:26:10.034726 9462 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 22:26:10.034835 9462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786227706282457-9923-0/minicluster-data/master-0/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786227706282457-9923-0/minicluster-data/master-0/wal
uuid: "9abb97d53c6d45bba360075b5fe344ad"
format_stamp: "Formatted at 2026-08-08 22:26:10 on dist-test-slave-ncp9"
server_key: "ad905cf36c979b9683c39a2dacf2a2ca"
server_key_iv: "fbf0a6e26a3ef1d9c505dc10c88026fc"
server_key_version: "encryptionkey@0"
I20260808 22:26:10.034926 9462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786227706282457-9923-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786227706282457-9923-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786227706282457-9923-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:26:10.046041 9462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:26:10.048700 9462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:26:10.048859 9462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:26:10.055398 9462 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35637
I20260808 22:26:10.055413 9567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35637 every 8 connection(s)
I20260808 22:26:10.056289 9462 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
I20260808 22:26:10.056924 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 9462
I20260808 22:26:10.057062 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786227706282457-9923-0/minicluster-data/master-0/wal/instance
I20260808 22:26:10.057328 9923 external_mini_cluster.cc:1428] Setting key 87ba76d946bdb1bca9e9b00786d888e0
I20260808 22:26:10.059324 9568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:26:10.065742 9568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9abb97d53c6d45bba360075b5fe344ad: Bootstrap starting.
Aug 08 22:26:10 dist-test-slave-ncp9 krb5kdc[9441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227969, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260808 22:26:10.068154 9568 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9abb97d53c6d45bba360075b5fe344ad: Neither blocks nor log segments found. Creating new log.
I20260808 22:26:10.069032 9568 log.cc:824] T 00000000000000000000000000000000 P 9abb97d53c6d45bba360075b5fe344ad: Log is configured to *not* fsync() on all Append() calls
I20260808 22:26:10.070885 9568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9abb97d53c6d45bba360075b5fe344ad: No bootstrap required, opened a new log
I20260808 22:26:10.071934 9571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:10.058692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.213:48791 (local address 127.0.0.1:35637)
0808 22:26:10.059090 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0808 22:26:10.059110 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0808 22:26:10.059150 (+ 40us) server_negotiation.cc:407] Connection header received
0808 22:26:10.059971 (+ 821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:26:10.059993 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:26:10.060403 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:26:10.060762 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:26:10.061440 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:10.062969 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:10.063834 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:10.064505 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:10.064678 (+ 173us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:26:10.067028 (+ 2350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:26:10.067034 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:26:10.067049 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:26:10.067079 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:26:10.068954 (+ 1875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:26:10.069441 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:26:10.069448 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:26:10.069453 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:26:10.069563 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:26:10.069913 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:26:10.069915 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:26:10.069917 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:26:10.070297 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:26:10.070486 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0808 22:26:10.070812 (+ 326us) server_negotiation.cc:299] Negotiation successful
0808 22:26:10.071039 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":109,"threads_started":1}
I20260808 22:26:10.074561 9923 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 22:26:10.074769 9568 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9abb97d53c6d45bba360075b5fe344ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9abb97d53c6d45bba360075b5fe344ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35637 } }
I20260808 22:26:10.074976 9568 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9abb97d53c6d45bba360075b5fe344ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:26:10.075029 9568 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9abb97d53c6d45bba360075b5fe344ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9abb97d53c6d45bba360075b5fe344ad, State: Initialized, Role: FOLLOWER
I20260808 22:26:10.075415 9568 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9abb97d53c6d45bba360075b5fe344ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9abb97d53c6d45bba360075b5fe344ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35637 } }
I20260808 22:26:10.075539 9568 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9abb97d53c6d45bba360075b5fe344ad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:26:10.075596 9568 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9abb97d53c6d45bba360075b5fe344ad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:26:10.075692 9568 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9abb97d53c6d45bba360075b5fe344ad [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:26:10.076614 9568 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9abb97d53c6d45bba360075b5fe344ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9abb97d53c6d45bba360075b5fe344ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35637 } }
I20260808 22:26:10.077003 9568 leader_election.cc:302] T 00000000000000000000000000000000 P 9abb97d53c6d45bba360075b5fe344ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9abb97d53c6d45bba360075b5fe344ad; no voters:
I20260808 22:26:10.077373 9568 leader_election.cc:288] T 00000000000000000000000000000000 P 9abb97d53c6d45bba360075b5fe344ad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:26:10.077448 9583 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9abb97d53c6d45bba360075b5fe344ad [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:26:10.077664 9583 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9abb97d53c6d45bba360075b5fe344ad [term 1 LEADER]: Becoming Leader. State: Replica: 9abb97d53c6d45bba360075b5fe344ad, State: Running, Role: LEADER
I20260808 22:26:10.078003 9583 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9abb97d53c6d45bba360075b5fe344ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9abb97d53c6d45bba360075b5fe344ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35637 } }
I20260808 22:26:10.078642 9568 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9abb97d53c6d45bba360075b5fe344ad [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:26:10.079015 9583 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9abb97d53c6d45bba360075b5fe344ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9abb97d53c6d45bba360075b5fe344ad. Latest consensus state: current_term: 1 leader_uuid: "9abb97d53c6d45bba360075b5fe344ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9abb97d53c6d45bba360075b5fe344ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35637 } } }
I20260808 22:26:10.079205 9583 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9abb97d53c6d45bba360075b5fe344ad [sys.catalog]: This master's current role is: LEADER
I20260808 22:26:10.079419 9584 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9abb97d53c6d45bba360075b5fe344ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9abb97d53c6d45bba360075b5fe344ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9abb97d53c6d45bba360075b5fe344ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35637 } } }
I20260808 22:26:10.079566 9584 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9abb97d53c6d45bba360075b5fe344ad [sys.catalog]: This master's current role is: LEADER
I20260808 22:26:10.079640 9590 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:26:10.083534 9590 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:26:10.085556 9571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:10.076479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.213:57709 (local address 127.0.0.1:35637)
0808 22:26:10.076574 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0808 22:26:10.076577 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:26:10.076749 (+ 172us) server_negotiation.cc:407] Connection header received
0808 22:26:10.076829 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:26:10.076831 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:26:10.076868 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:26:10.076956 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:26:10.077713 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:10.078458 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:10.079652 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:10.079987 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:10.080064 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:26:10.081909 (+ 1845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:26:10.081912 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:26:10.081915 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:26:10.081939 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:26:10.083498 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:26:10.084192 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:26:10.084195 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:26:10.084197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:26:10.084283 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:26:10.084632 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:26:10.084635 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:26:10.084637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:26:10.084843 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:26:10.085059 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0808 22:26:10.085318 (+ 259us) server_negotiation.cc:299] Negotiation successful
0808 22:26:10.085414 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260808 22:26:10.089066 9590 catalog_manager.cc:1347] Generated new cluster ID: 8d084a4319514311ae86aae46f026931
I20260808 22:26:10.089119 9590 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:26:10.120640 9590 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:26:10.121984 9590 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:26:10.126217 9590 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9abb97d53c6d45bba360075b5fe344ad: Generated new TSK 0
I20260808 22:26:10.126979 9590 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 22:26:10.131907 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 9462
W20260808 22:26:10.145574 9437 connection.cc:535] client connection to 127.0.0.1:35637 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35637): Connection reset by peer (error 104)
2026-08-08T22:26:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1478 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786227706282457-9923-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 22:26:10 dist-test-slave-ncp9 krb5kdc[9610](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 22:26:10 dist-test-slave-ncp9 krb5kdc[9610](info): set up 2 sockets
Aug 08 22:26:10 dist-test-slave-ncp9 krb5kdc[9610](info): commencing operation
krb5kdc: starting...
W20260808 22:26:11.250862 9923 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786227706282457-9923-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786227706282457-9923-0/krb5kdc/test-user.keytab.
Aug 08 22:26:11 dist-test-slave-ncp9 krb5kdc[9610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T22:26:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:26:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786227706282457-9923-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786227706282457-9923-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 22:26:11.365238 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786227706282457-9923-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786227706282457-9923-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786227706282457-9923-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44161
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786227706282457-9923-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44161
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:26:11.474180 9628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:26:11.474419 9628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:26:11.474462 9628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:26:11.478415 9628 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:26:11.478469 9628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:26:11.478487 9628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:26:11.478502 9628 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:26:11.478515 9628 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:26:11.483644 9628 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786227706282457-9923-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786227706282457-9923-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44161
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786227706282457-9923-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9628
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786227706282457-9923-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:26:11.484614 9628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:26:11.485589 9628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 22:26:11.494014 9628 server_base.cc:1034] running on GCE node
W20260808 22:26:11.494084 9636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:26:11.494047 9637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:26:11.494026 9640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:26:11.494844 9628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:26:11.495970 9628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:26:11.497126 9628 hybrid_clock.cc:648] HybridClock initialized: now 1786227971497110 us; error 33 us; skew 500 ppm
Aug 08 22:26:11 dist-test-slave-ncp9 krb5kdc[9610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227971, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:26:11.499810 9628 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 22:26:11.500749 9628 webserver.cc:460] Webserver started at http://127.0.0.1:42615/ using document root <none> and password file <none>
I20260808 22:26:11.501317 9628 fs_manager.cc:359] Metadata directory not provided
I20260808 22:26:11.501387 9628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:26:11.501605 9628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:26:11.503665 9628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786227706282457-9923-0/minicluster-data/master-0/data/instance:
uuid: "c5e7edb6d0374b098d0666b35e45bec2"
format_stamp: "Formatted at 2026-08-08 22:26:11 on dist-test-slave-ncp9"
server_key: "67ab5c3d9860b94ceafe9cff6b42ae6e"
server_key_iv: "656a50f49d458cea2eaed83406f4ce49"
server_key_version: "encryptionkey@0"
I20260808 22:26:11.504200 9628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786227706282457-9923-0/minicluster-data/master-0/wal/instance:
uuid: "c5e7edb6d0374b098d0666b35e45bec2"
format_stamp: "Formatted at 2026-08-08 22:26:11 on dist-test-slave-ncp9"
server_key: "67ab5c3d9860b94ceafe9cff6b42ae6e"
server_key_iv: "656a50f49d458cea2eaed83406f4ce49"
server_key_version: "encryptionkey@0"
I20260808 22:26:11.507817 9628 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260808 22:26:11.510290 9650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:26:11.511503 9628 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 22:26:11.511648 9628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786227706282457-9923-0/minicluster-data/master-0/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786227706282457-9923-0/minicluster-data/master-0/wal
uuid: "c5e7edb6d0374b098d0666b35e45bec2"
format_stamp: "Formatted at 2026-08-08 22:26:11 on dist-test-slave-ncp9"
server_key: "67ab5c3d9860b94ceafe9cff6b42ae6e"
server_key_iv: "656a50f49d458cea2eaed83406f4ce49"
server_key_version: "encryptionkey@0"
I20260808 22:26:11.511746 9628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786227706282457-9923-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786227706282457-9923-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786227706282457-9923-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:26:11.531906 9628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:26:11.534572 9628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:26:11.534745 9628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:26:11.541787 9628 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44161
I20260808 22:26:11.541805 9727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44161 every 8 connection(s)
I20260808 22:26:11.542723 9628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
I20260808 22:26:11.545434 9728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:26:11.551065 9728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5e7edb6d0374b098d0666b35e45bec2: Bootstrap starting.
I20260808 22:26:11.551292 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 9628
I20260808 22:26:11.551409 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786227706282457-9923-0/minicluster-data/master-0/wal/instance
I20260808 22:26:11.551672 9923 external_mini_cluster.cc:1428] Setting key 4d817617b24a9366c0d4b6d541688444
I20260808 22:26:11.553611 9728 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c5e7edb6d0374b098d0666b35e45bec2: Neither blocks nor log segments found. Creating new log.
I20260808 22:26:11.554337 9728 log.cc:824] T 00000000000000000000000000000000 P c5e7edb6d0374b098d0666b35e45bec2: Log is configured to *not* fsync() on all Append() calls
I20260808 22:26:11.556521 9728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5e7edb6d0374b098d0666b35e45bec2: No bootstrap required, opened a new log
I20260808 22:26:11.560792 9728 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5e7edb6d0374b098d0666b35e45bec2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5e7edb6d0374b098d0666b35e45bec2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44161 } }
I20260808 22:26:11.561093 9728 raft_consensus.cc:374] T 00000000000000000000000000000000 P c5e7edb6d0374b098d0666b35e45bec2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:26:11.561162 9728 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5e7edb6d0374b098d0666b35e45bec2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e7edb6d0374b098d0666b35e45bec2, State: Initialized, Role: FOLLOWER
I20260808 22:26:11.561167 9734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:11.553030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.213:57751 (local address 127.0.0.1:44161)
0808 22:26:11.553441 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0808 22:26:11.553457 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 22:26:11.553491 (+ 34us) server_negotiation.cc:407] Connection header received
0808 22:26:11.554140 (+ 649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:26:11.554159 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:26:11.554515 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:26:11.554801 (+ 286us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:26:11.555145 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:11.556577 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:11.557502 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:11.558194 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:11.558357 (+ 163us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:26:11.559154 (+ 797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:26:11.559162 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:26:11.559181 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:26:11.559717 (+ 536us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:26:11.559785 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:26:11.559814 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:26:11.559947 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0808 22:26:11.560041 (+ 94us) server_negotiation.cc:299] Negotiation successful
0808 22:26:11.560175 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":109,"threads_started":1}
I20260808 22:26:11.561616 9728 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5e7edb6d0374b098d0666b35e45bec2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5e7edb6d0374b098d0666b35e45bec2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44161 } }
I20260808 22:26:11.561754 9728 raft_consensus.cc:388] T 00000000000000000000000000000000 P c5e7edb6d0374b098d0666b35e45bec2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:26:11.561820 9728 raft_consensus.cc:482] T 00000000000000000000000000000000 P c5e7edb6d0374b098d0666b35e45bec2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:26:11.561925 9728 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5e7edb6d0374b098d0666b35e45bec2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:26:11.562800 9728 raft_consensus.cc:504] T 00000000000000000000000000000000 P c5e7edb6d0374b098d0666b35e45bec2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5e7edb6d0374b098d0666b35e45bec2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44161 } }
I20260808 22:26:11.563160 9728 leader_election.cc:302] T 00000000000000000000000000000000 P c5e7edb6d0374b098d0666b35e45bec2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c5e7edb6d0374b098d0666b35e45bec2; no voters:
I20260808 22:26:11.563535 9728 leader_election.cc:288] T 00000000000000000000000000000000 P c5e7edb6d0374b098d0666b35e45bec2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:26:11.563616 9736 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c5e7edb6d0374b098d0666b35e45bec2 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:26:11.563834 9736 raft_consensus.cc:686] T 00000000000000000000000000000000 P c5e7edb6d0374b098d0666b35e45bec2 [term 1 LEADER]: Becoming Leader. State: Replica: c5e7edb6d0374b098d0666b35e45bec2, State: Running, Role: LEADER
I20260808 22:26:11.564047 9923 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 22:26:11.564190 9736 consensus_queue.cc:237] T 00000000000000000000000000000000 P c5e7edb6d0374b098d0666b35e45bec2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5e7edb6d0374b098d0666b35e45bec2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44161 } }
I20260808 22:26:11.564836 9728 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5e7edb6d0374b098d0666b35e45bec2 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:26:11.565732 9738 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5e7edb6d0374b098d0666b35e45bec2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c5e7edb6d0374b098d0666b35e45bec2. Latest consensus state: current_term: 1 leader_uuid: "c5e7edb6d0374b098d0666b35e45bec2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5e7edb6d0374b098d0666b35e45bec2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44161 } } }
I20260808 22:26:11.565876 9738 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5e7edb6d0374b098d0666b35e45bec2 [sys.catalog]: This master's current role is: LEADER
I20260808 22:26:11.565732 9737 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5e7edb6d0374b098d0666b35e45bec2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c5e7edb6d0374b098d0666b35e45bec2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5e7edb6d0374b098d0666b35e45bec2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44161 } } }
I20260808 22:26:11.566072 9737 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5e7edb6d0374b098d0666b35e45bec2 [sys.catalog]: This master's current role is: LEADER
W20260808 22:26:11.569538 9763 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c5e7edb6d0374b098d0666b35e45bec2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 22:26:11.569605 9763 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 22:26:11.569702 9752 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:26:11.570497 9752 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:26:11.571507 9734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:11.566603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.213:50487 (local address 127.0.0.1:44161)
0808 22:26:11.566706 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0808 22:26:11.566711 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:26:11.567148 (+ 437us) server_negotiation.cc:407] Connection header received
0808 22:26:11.567381 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:26:11.567384 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:26:11.567430 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:26:11.567477 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:26:11.567815 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:11.568559 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:11.569738 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:11.570055 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:11.570162 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:26:11.571047 (+ 885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:26:11.571050 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:26:11.571053 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:26:11.571066 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:26:11.571093 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:26:11.571106 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:26:11.571195 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0808 22:26:11.571354 (+ 159us) server_negotiation.cc:299] Negotiation successful
0808 22:26:11.571388 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260808 22:26:11.575625 9752 catalog_manager.cc:1347] Generated new cluster ID: 8b6aaf70f5e9486a803e2c033f4e946c
I20260808 22:26:11.575690 9752 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:26:11.585578 9752 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:26:11.586426 9752 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:26:11.592561 9752 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c5e7edb6d0374b098d0666b35e45bec2: Generated new TSK 0
I20260808 22:26:11.593214 9752 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 22:26:11.604737 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 9628
W20260808 22:26:11.621114 9607 connection.cc:535] client connection to 127.0.0.1:44161 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44161): Connection reset by peer (error 104)
2026-08-08T22:26:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1475 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786227706282457-9923-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 22:26:11 dist-test-slave-ncp9 krb5kdc[9774](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 22:26:11 dist-test-slave-ncp9 krb5kdc[9774](info): set up 2 sockets
Aug 08 22:26:11 dist-test-slave-ncp9 krb5kdc[9774](info): commencing operation
krb5kdc: starting...
W20260808 22:26:12.727530 9923 mini_kdc.cc:121] Time spent starting KDC: real 1.092s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786227706282457-9923-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786227706282457-9923-0/krb5kdc/test-user.keytab.
Aug 08 22:26:12 dist-test-slave-ncp9 krb5kdc[9774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227972, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T22:26:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:26:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786227706282457-9923-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786227706282457-9923-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 22:26:12.841549 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786227706282457-9923-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786227706282457-9923-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786227706282457-9923-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35493
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786227706282457-9923-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35493
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:26:12.949451 9790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:26:12.949699 9790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:26:12.949754 9790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:26:12.953634 9790 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:26:12.953684 9790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:26:12.953711 9790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:26:12.953733 9790 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:26:12.953758 9790 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:26:12.958834 9790 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786227706282457-9923-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786227706282457-9923-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35493
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786227706282457-9923-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9790
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786227706282457-9923-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:26:12.959782 9790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:26:12.960736 9790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 22:26:12.968802 9790 server_base.cc:1034] running on GCE node
W20260808 22:26:12.968773 9797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:26:12.968722 9798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:26:12.968729 9800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:26:12.969545 9790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:26:12.970530 9790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:26:12.971689 9790 hybrid_clock.cc:648] HybridClock initialized: now 1786227972971677 us; error 24 us; skew 500 ppm
Aug 08 22:26:12 dist-test-slave-ncp9 krb5kdc[9774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227972, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:26:12.978396 9790 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 22:26:12.979205 9790 webserver.cc:460] Webserver started at http://127.0.0.1:44689/ using document root <none> and password file <none>
I20260808 22:26:12.979694 9790 fs_manager.cc:359] Metadata directory not provided
I20260808 22:26:12.979750 9790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:26:12.979907 9790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:26:12.981851 9790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786227706282457-9923-0/minicluster-data/master-0/data/instance:
uuid: "10cab0bd169740f59ccf4ff299db75d7"
format_stamp: "Formatted at 2026-08-08 22:26:12 on dist-test-slave-ncp9"
server_key: "9f6b828b3c21cfa7a17a48fa5721c470"
server_key_iv: "d26f63a1a7123564cee80bb757462b6f"
server_key_version: "encryptionkey@0"
I20260808 22:26:12.982359 9790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786227706282457-9923-0/minicluster-data/master-0/wal/instance:
uuid: "10cab0bd169740f59ccf4ff299db75d7"
format_stamp: "Formatted at 2026-08-08 22:26:12 on dist-test-slave-ncp9"
server_key: "9f6b828b3c21cfa7a17a48fa5721c470"
server_key_iv: "d26f63a1a7123564cee80bb757462b6f"
server_key_version: "encryptionkey@0"
I20260808 22:26:12.985764 9790 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 22:26:12.987937 9807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:26:12.989095 9790 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 22:26:12.989207 9790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786227706282457-9923-0/minicluster-data/master-0/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786227706282457-9923-0/minicluster-data/master-0/wal
uuid: "10cab0bd169740f59ccf4ff299db75d7"
format_stamp: "Formatted at 2026-08-08 22:26:12 on dist-test-slave-ncp9"
server_key: "9f6b828b3c21cfa7a17a48fa5721c470"
server_key_iv: "d26f63a1a7123564cee80bb757462b6f"
server_key_version: "encryptionkey@0"
I20260808 22:26:12.989291 9790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786227706282457-9923-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786227706282457-9923-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786227706282457-9923-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:26:12.993502 9790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:26:12.995945 9790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:26:12.996100 9790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:26:13.002686 9790 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35493
I20260808 22:26:13.002696 9879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35493 every 8 connection(s)
I20260808 22:26:13.003510 9790 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
I20260808 22:26:13.006138 9880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:26:13.006618 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 9790
I20260808 22:26:13.006760 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786227706282457-9923-0/minicluster-data/master-0/wal/instance
I20260808 22:26:13.007062 9923 external_mini_cluster.cc:1428] Setting key b541a8a1160be58d8b5062d07d0bee5a
I20260808 22:26:13.011950 9880 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10cab0bd169740f59ccf4ff299db75d7: Bootstrap starting.
I20260808 22:26:13.012197 9886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:13.008330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.213:42039 (local address 127.0.0.1:35493)
0808 22:26:13.008753 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0808 22:26:13.008772 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0808 22:26:13.008991 (+ 219us) server_negotiation.cc:407] Connection header received
0808 22:26:13.009719 (+ 728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:26:13.009736 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:26:13.010011 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:26:13.010307 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:26:13.011059 (+ 752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:26:13.011064 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:26:13.011075 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:26:13.011085 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:26:13.011138 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:26:13.011156 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:26:13.011279 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0808 22:26:13.011350 (+ 71us) server_negotiation.cc:299] Negotiation successful
0808 22:26:13.011504 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":101,"threads_started":1}
I20260808 22:26:13.014320 9880 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 10cab0bd169740f59ccf4ff299db75d7: Neither blocks nor log segments found. Creating new log.
I20260808 22:26:13.014911 9923 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 22:26:13.015152 9880 log.cc:824] T 00000000000000000000000000000000 P 10cab0bd169740f59ccf4ff299db75d7: Log is configured to *not* fsync() on all Append() calls
I20260808 22:26:13.017156 9880 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10cab0bd169740f59ccf4ff299db75d7: No bootstrap required, opened a new log
I20260808 22:26:13.018306 9886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:13.016834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.213:48967 (local address 127.0.0.1:35493)
0808 22:26:13.016949 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0808 22:26:13.016953 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:26:13.017123 (+ 170us) server_negotiation.cc:407] Connection header received
0808 22:26:13.017229 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:26:13.017232 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:26:13.017263 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:26:13.017333 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:26:13.017970 (+ 637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:26:13.017974 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:26:13.017976 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:26:13.017987 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:26:13.018012 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:26:13.018022 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:26:13.018097 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0808 22:26:13.018164 (+ 67us) server_negotiation.cc:299] Negotiation successful
0808 22:26:13.018200 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260808 22:26:13.020992 9880 raft_consensus.cc:348] T 00000000000000000000000000000000 P 10cab0bd169740f59ccf4ff299db75d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10cab0bd169740f59ccf4ff299db75d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35493 } }
I20260808 22:26:13.021204 9880 raft_consensus.cc:374] T 00000000000000000000000000000000 P 10cab0bd169740f59ccf4ff299db75d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:26:13.021255 9880 raft_consensus.cc:729] T 00000000000000000000000000000000 P 10cab0bd169740f59ccf4ff299db75d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10cab0bd169740f59ccf4ff299db75d7, State: Initialized, Role: FOLLOWER
I20260808 22:26:13.021651 9880 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10cab0bd169740f59ccf4ff299db75d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10cab0bd169740f59ccf4ff299db75d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35493 } }
I20260808 22:26:13.021775 9880 raft_consensus.cc:388] T 00000000000000000000000000000000 P 10cab0bd169740f59ccf4ff299db75d7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:26:13.021837 9880 raft_consensus.cc:482] T 00000000000000000000000000000000 P 10cab0bd169740f59ccf4ff299db75d7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:26:13.021926 9880 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 10cab0bd169740f59ccf4ff299db75d7 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:26:13.022789 9880 raft_consensus.cc:504] T 00000000000000000000000000000000 P 10cab0bd169740f59ccf4ff299db75d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10cab0bd169740f59ccf4ff299db75d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35493 } }
I20260808 22:26:13.023098 9880 leader_election.cc:302] T 00000000000000000000000000000000 P 10cab0bd169740f59ccf4ff299db75d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 10cab0bd169740f59ccf4ff299db75d7; no voters:
I20260808 22:26:13.023358 9880 leader_election.cc:288] T 00000000000000000000000000000000 P 10cab0bd169740f59ccf4ff299db75d7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:26:13.023461 9897 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 10cab0bd169740f59ccf4ff299db75d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:26:13.023697 9897 raft_consensus.cc:686] T 00000000000000000000000000000000 P 10cab0bd169740f59ccf4ff299db75d7 [term 1 LEADER]: Becoming Leader. State: Replica: 10cab0bd169740f59ccf4ff299db75d7, State: Running, Role: LEADER
I20260808 22:26:13.024060 9897 consensus_queue.cc:237] T 00000000000000000000000000000000 P 10cab0bd169740f59ccf4ff299db75d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10cab0bd169740f59ccf4ff299db75d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35493 } }
I20260808 22:26:13.024520 9880 sys_catalog.cc:564] T 00000000000000000000000000000000 P 10cab0bd169740f59ccf4ff299db75d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:26:13.025607 9899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10cab0bd169740f59ccf4ff299db75d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10cab0bd169740f59ccf4ff299db75d7. Latest consensus state: current_term: 1 leader_uuid: "10cab0bd169740f59ccf4ff299db75d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10cab0bd169740f59ccf4ff299db75d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35493 } } }
I20260808 22:26:13.025770 9899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10cab0bd169740f59ccf4ff299db75d7 [sys.catalog]: This master's current role is: LEADER
I20260808 22:26:13.025650 9898 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10cab0bd169740f59ccf4ff299db75d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "10cab0bd169740f59ccf4ff299db75d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10cab0bd169740f59ccf4ff299db75d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35493 } } }
I20260808 22:26:13.026055 9898 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10cab0bd169740f59ccf4ff299db75d7 [sys.catalog]: This master's current role is: LEADER
I20260808 22:26:13.026166 9909 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:26:13.028976 9909 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:26:13.034175 9909 catalog_manager.cc:1347] Generated new cluster ID: fd3e954ccf014579943f5e823049330a
I20260808 22:26:13.034250 9909 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:26:13.041612 9909 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:26:13.041921 9909 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:26:13.050666 9909 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 10cab0bd169740f59ccf4ff299db75d7: Generated new TSK 0
I20260808 22:26:13.051405 9909 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 22:26:13.060585 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 9790
2026-08-08T22:26:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1447 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786227706282457-9923-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 22:26:13 dist-test-slave-ncp9 krb5kdc[9936](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 22:26:13 dist-test-slave-ncp9 krb5kdc[9936](info): set up 2 sockets
Aug 08 22:26:13 dist-test-slave-ncp9 krb5kdc[9936](info): commencing operation
krb5kdc: starting...
W20260808 22:26:14.181293 9923 mini_kdc.cc:121] Time spent starting KDC: real 1.099s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786227706282457-9923-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786227706282457-9923-0/krb5kdc/test-user.keytab.
Aug 08 22:26:14 dist-test-slave-ncp9 krb5kdc[9936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227974, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T22:26:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:26:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.176.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.176.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.176.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab.
Entry for principal kudu/127.9.176.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab.
I20260808 22:26:14.297956 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786227706282457-9923-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786227706282457-9923-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786227706282457-9923-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:40675
--webserver_interface=127.9.176.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab
--principal=kudu/127.9.176.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:40701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.176.254:40675
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:26:14.406015 9953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:26:14.406266 9953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:26:14.406320 9953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:26:14.410226 9953 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:26:14.410286 9953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:26:14.410315 9953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:26:14.410334 9953 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:26:14.410360 9953 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:26:14.415632 9953 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:40701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786227706282457-9923-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786227706282457-9923-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.176.254:40675
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab
--principal=kudu/127.9.176.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:40675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.176.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9953
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786227706282457-9923-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:26:14.416678 9953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:26:14.417708 9953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:26:14.426105 9960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:26:14.426173 9963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:26:14.426345 9953 server_base.cc:1034] running on GCE node
W20260808 22:26:14.426173 9961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:26:14.427028 9953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:26:14.428074 9953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:26:14.429234 9953 hybrid_clock.cc:648] HybridClock initialized: now 1786227974429214 us; error 39 us; skew 500 ppm
Aug 08 22:26:14 dist-test-slave-ncp9 krb5kdc[9936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227974, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:26:14.431823 9953 init.cc:377] Logged in from keytab as kudu/127.9.176.254@KRBTEST.COM (short username kudu)
I20260808 22:26:14.432643 9953 webserver.cc:460] Webserver started at http://127.9.176.254:36151/ using document root <none> and password file <none>
I20260808 22:26:14.433185 9953 fs_manager.cc:359] Metadata directory not provided
I20260808 22:26:14.433254 9953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:26:14.433450 9953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:26:14.435402 9953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786227706282457-9923-0/minicluster-data/master-0/data/instance:
uuid: "b0819b8e344d45baad994a7cb1baad25"
format_stamp: "Formatted at 2026-08-08 22:26:14 on dist-test-slave-ncp9"
server_key: "dc6b4f9ec5a675e763d54cf14d168aeb"
server_key_iv: "c670b7e491d798dd33b7d45c62bd5217"
server_key_version: "encryptionkey@0"
I20260808 22:26:14.435940 9953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786227706282457-9923-0/minicluster-data/master-0/wal/instance:
uuid: "b0819b8e344d45baad994a7cb1baad25"
format_stamp: "Formatted at 2026-08-08 22:26:14 on dist-test-slave-ncp9"
server_key: "dc6b4f9ec5a675e763d54cf14d168aeb"
server_key_iv: "c670b7e491d798dd33b7d45c62bd5217"
server_key_version: "encryptionkey@0"
I20260808 22:26:14.439493 9953 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260808 22:26:14.441836 9972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:26:14.443019 9953 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 22:26:14.443131 9953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786227706282457-9923-0/minicluster-data/master-0/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786227706282457-9923-0/minicluster-data/master-0/wal
uuid: "b0819b8e344d45baad994a7cb1baad25"
format_stamp: "Formatted at 2026-08-08 22:26:14 on dist-test-slave-ncp9"
server_key: "dc6b4f9ec5a675e763d54cf14d168aeb"
server_key_iv: "c670b7e491d798dd33b7d45c62bd5217"
server_key_version: "encryptionkey@0"
I20260808 22:26:14.443219 9953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786227706282457-9923-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786227706282457-9923-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786227706282457-9923-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:26:14.458927 9953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:26:14.461768 9953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:26:14.461937 9953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:26:14.468565 9953 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.254:40675
I20260808 22:26:14.468587 10046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.254:40675 every 8 connection(s)
I20260808 22:26:14.469476 9953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
I20260808 22:26:14.472132 10047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:26:14.473881 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 9953
I20260808 22:26:14.474037 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786227706282457-9923-0/minicluster-data/master-0/wal/instance
I20260808 22:26:14.474326 9923 external_mini_cluster.cc:1428] Setting key f64165b4ef8c5fcd49ff66db673ca0c1
I20260808 22:26:14.478760 10047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0819b8e344d45baad994a7cb1baad25: Bootstrap starting.
I20260808 22:26:14.481303 10047 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b0819b8e344d45baad994a7cb1baad25: Neither blocks nor log segments found. Creating new log.
I20260808 22:26:14.482175 10047 log.cc:824] T 00000000000000000000000000000000 P b0819b8e344d45baad994a7cb1baad25: Log is configured to *not* fsync() on all Append() calls
Aug 08 22:26:14 dist-test-slave-ncp9 krb5kdc[9936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227974, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
I20260808 22:26:14.484273 10047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0819b8e344d45baad994a7cb1baad25: No bootstrap required, opened a new log
I20260808 22:26:14.488209 10047 raft_consensus.cc:348] T 00000000000000000000000000000000 P b0819b8e344d45baad994a7cb1baad25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0819b8e344d45baad994a7cb1baad25" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 40675 } }
I20260808 22:26:14.488418 10047 raft_consensus.cc:374] T 00000000000000000000000000000000 P b0819b8e344d45baad994a7cb1baad25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:26:14.488453 10047 raft_consensus.cc:729] T 00000000000000000000000000000000 P b0819b8e344d45baad994a7cb1baad25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0819b8e344d45baad994a7cb1baad25, State: Initialized, Role: FOLLOWER
I20260808 22:26:14.488849 10052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:14.475651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54017 (local address 127.9.176.254:40675)
0808 22:26:14.476070 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0808 22:26:14.476089 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0808 22:26:14.476132 (+ 43us) server_negotiation.cc:407] Connection header received
0808 22:26:14.476951 (+ 819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:26:14.476975 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:26:14.477405 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:26:14.477757 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:26:14.478440 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:14.479868 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:14.480755 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:14.481468 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:14.481633 (+ 165us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:26:14.484290 (+ 2657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:26:14.484295 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:26:14.484307 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:26:14.484339 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:26:14.485978 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:26:14.486571 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:26:14.486576 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:26:14.486580 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:26:14.486693 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:26:14.487010 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:26:14.487011 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:26:14.487013 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:26:14.487378 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:26:14.487539 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0808 22:26:14.487841 (+ 302us) server_negotiation.cc:299] Negotiation successful
0808 22:26:14.488054 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":118,"threads_started":1}
I20260808 22:26:14.488880 10047 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0819b8e344d45baad994a7cb1baad25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0819b8e344d45baad994a7cb1baad25" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 40675 } }
I20260808 22:26:14.489043 10047 raft_consensus.cc:388] T 00000000000000000000000000000000 P b0819b8e344d45baad994a7cb1baad25 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:26:14.489130 10047 raft_consensus.cc:482] T 00000000000000000000000000000000 P b0819b8e344d45baad994a7cb1baad25 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:26:14.489230 10047 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b0819b8e344d45baad994a7cb1baad25 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:26:14.490159 10047 raft_consensus.cc:504] T 00000000000000000000000000000000 P b0819b8e344d45baad994a7cb1baad25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0819b8e344d45baad994a7cb1baad25" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 40675 } }
I20260808 22:26:14.490507 10047 leader_election.cc:302] T 00000000000000000000000000000000 P b0819b8e344d45baad994a7cb1baad25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b0819b8e344d45baad994a7cb1baad25; no voters:
I20260808 22:26:14.490816 10047 leader_election.cc:288] T 00000000000000000000000000000000 P b0819b8e344d45baad994a7cb1baad25 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:26:14.490911 10055 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b0819b8e344d45baad994a7cb1baad25 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:26:14.491132 10055 raft_consensus.cc:686] T 00000000000000000000000000000000 P b0819b8e344d45baad994a7cb1baad25 [term 1 LEADER]: Becoming Leader. State: Replica: b0819b8e344d45baad994a7cb1baad25, State: Running, Role: LEADER
I20260808 22:26:14.491446 10055 consensus_queue.cc:237] T 00000000000000000000000000000000 P b0819b8e344d45baad994a7cb1baad25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0819b8e344d45baad994a7cb1baad25" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 40675 } }
I20260808 22:26:14.491878 9923 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 22:26:14.492108 10047 sys_catalog.cc:564] T 00000000000000000000000000000000 P b0819b8e344d45baad994a7cb1baad25 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:26:14.492980 10057 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0819b8e344d45baad994a7cb1baad25 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b0819b8e344d45baad994a7cb1baad25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0819b8e344d45baad994a7cb1baad25" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 40675 } } }
I20260808 22:26:14.493120 10057 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0819b8e344d45baad994a7cb1baad25 [sys.catalog]: This master's current role is: LEADER
I20260808 22:26:14.493468 10056 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0819b8e344d45baad994a7cb1baad25 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0819b8e344d45baad994a7cb1baad25. Latest consensus state: current_term: 1 leader_uuid: "b0819b8e344d45baad994a7cb1baad25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0819b8e344d45baad994a7cb1baad25" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 40675 } } }
I20260808 22:26:14.493547 10056 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0819b8e344d45baad994a7cb1baad25 [sys.catalog]: This master's current role is: LEADER
W20260808 22:26:14.497138 10085 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b0819b8e344d45baad994a7cb1baad25: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 22:26:14.497224 10085 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 22:26:14.497290 10072 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:26:14.497965 10072 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:26:14.502147 10052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:14.494131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50915 (local address 127.9.176.254:40675)
0808 22:26:14.494223 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0808 22:26:14.494227 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:26:14.494690 (+ 463us) server_negotiation.cc:407] Connection header received
0808 22:26:14.494837 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:26:14.494840 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:26:14.494885 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:26:14.494951 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:26:14.495600 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:14.496124 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:14.496943 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:14.497176 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:14.497263 (+ 87us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:26:14.498616 (+ 1353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:26:14.498620 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:26:14.498622 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:26:14.498644 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:26:14.500077 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:26:14.500643 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:26:14.500647 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:26:14.500648 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:26:14.500733 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:26:14.501155 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:26:14.501159 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:26:14.501160 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:26:14.501364 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:26:14.501472 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0808 22:26:14.501913 (+ 441us) server_negotiation.cc:299] Negotiation successful
0808 22:26:14.502003 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260808 22:26:14.503875 10072 catalog_manager.cc:1347] Generated new cluster ID: c0a2feba4832423f8a4dc1071a8dee29
I20260808 22:26:14.503942 10072 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:26:14.525336 10072 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:26:14.526358 10072 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:26:14.535281 10072 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b0819b8e344d45baad994a7cb1baad25: Generated new TSK 0
I20260808 22:26:14.536002 10072 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 22:26:14.540227 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 9953
W20260808 22:26:14.550802 9933 connection.cc:535] client connection to 127.9.176.254:40675 recv error: Network error: failed to read from TLS socket (remote: 127.9.176.254:40675): Connection reset by peer (error 104)
2026-08-08T22:26:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1481 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786227706282457-9923-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 22:26:14 dist-test-slave-ncp9 krb5kdc[10095](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 22:26:14 dist-test-slave-ncp9 krb5kdc[10095](info): set up 2 sockets
Aug 08 22:26:14 dist-test-slave-ncp9 krb5kdc[10095](info): commencing operation
krb5kdc: starting...
W20260808 22:26:15.665875 9923 mini_kdc.cc:121] Time spent starting KDC: real 1.101s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786227706282457-9923-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786227706282457-9923-0/krb5kdc/test-user.keytab.
Aug 08 22:26:15 dist-test-slave-ncp9 krb5kdc[10095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227975, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T22:26:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:26:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.176.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.176.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.176.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab.
Entry for principal kudu/127.9.176.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab.
I20260808 22:26:15.785005 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786227706282457-9923-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786227706282457-9923-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786227706282457-9923-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:34397
--webserver_interface=127.9.176.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab
--principal=kudu/127.9.176.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:43363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.176.254:34397
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:26:15.894289 10115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:26:15.894531 10115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:26:15.894572 10115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:26:15.898414 10115 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:26:15.898473 10115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:26:15.898490 10115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:26:15.898506 10115 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:26:15.898519 10115 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:26:15.903571 10115 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:43363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786227706282457-9923-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786227706282457-9923-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.176.254:34397
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab
--principal=kudu/127.9.176.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:34397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.176.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10115
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786227706282457-9923-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:26:15.904505 10115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:26:15.905478 10115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:26:15.913779 10121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:26:15.913769 10122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:26:15.913841 10115 server_base.cc:1034] running on GCE node
W20260808 22:26:15.913769 10126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:26:15.914566 10115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:26:15.915637 10115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:26:15.916792 10115 hybrid_clock.cc:648] HybridClock initialized: now 1786227975916777 us; error 28 us; skew 500 ppm
Aug 08 22:26:15 dist-test-slave-ncp9 krb5kdc[10095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227975, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:26:15.919538 10115 init.cc:377] Logged in from keytab as kudu/127.9.176.254@KRBTEST.COM (short username kudu)
I20260808 22:26:15.920411 10115 webserver.cc:460] Webserver started at http://127.9.176.254:36021/ using document root <none> and password file <none>
I20260808 22:26:15.920951 10115 fs_manager.cc:359] Metadata directory not provided
I20260808 22:26:15.921018 10115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:26:15.921188 10115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:26:15.923269 10115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786227706282457-9923-0/minicluster-data/master-0/data/instance:
uuid: "1d062e632fd74969ac83283ec3a91a56"
format_stamp: "Formatted at 2026-08-08 22:26:15 on dist-test-slave-ncp9"
server_key: "ece9510e397dec60bf2a3ff944c32379"
server_key_iv: "8cde14e4913371f8bed8f93233e99ae1"
server_key_version: "encryptionkey@0"
I20260808 22:26:15.923784 10115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786227706282457-9923-0/minicluster-data/master-0/wal/instance:
uuid: "1d062e632fd74969ac83283ec3a91a56"
format_stamp: "Formatted at 2026-08-08 22:26:15 on dist-test-slave-ncp9"
server_key: "ece9510e397dec60bf2a3ff944c32379"
server_key_iv: "8cde14e4913371f8bed8f93233e99ae1"
server_key_version: "encryptionkey@0"
I20260808 22:26:15.927342 10115 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 22:26:15.929699 10138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:26:15.930902 10115 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 22:26:15.931005 10115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786227706282457-9923-0/minicluster-data/master-0/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786227706282457-9923-0/minicluster-data/master-0/wal
uuid: "1d062e632fd74969ac83283ec3a91a56"
format_stamp: "Formatted at 2026-08-08 22:26:15 on dist-test-slave-ncp9"
server_key: "ece9510e397dec60bf2a3ff944c32379"
server_key_iv: "8cde14e4913371f8bed8f93233e99ae1"
server_key_version: "encryptionkey@0"
I20260808 22:26:15.931106 10115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786227706282457-9923-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786227706282457-9923-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786227706282457-9923-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:26:15.939949 10115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:26:15.942706 10115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:26:15.942864 10115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:26:15.949400 10115 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.254:34397
I20260808 22:26:15.949434 10214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.254:34397 every 8 connection(s)
I20260808 22:26:15.950295 10115 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
I20260808 22:26:15.950707 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 10115
I20260808 22:26:15.950881 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786227706282457-9923-0/minicluster-data/master-0/wal/instance
I20260808 22:26:15.951155 9923 external_mini_cluster.cc:1428] Setting key c6c37b241357c64a950015d36ee90953
I20260808 22:26:15.953361 10215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:26:15.958863 10215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d062e632fd74969ac83283ec3a91a56: Bootstrap starting.
I20260808 22:26:15.959698 10219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:15.952465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38323 (local address 127.9.176.254:34397)
0808 22:26:15.952956 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0808 22:26:15.952975 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0808 22:26:15.953013 (+ 38us) server_negotiation.cc:407] Connection header received
0808 22:26:15.953692 (+ 679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:26:15.953710 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:26:15.954040 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:26:15.954335 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:26:15.954718 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:15.955705 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:15.956584 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:15.957099 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:15.957255 (+ 156us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:26:15.958136 (+ 881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:26:15.958140 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:26:15.958151 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:26:15.958506 (+ 355us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:26:15.958559 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:26:15.958581 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:26:15.958689 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0808 22:26:15.958877 (+ 188us) server_negotiation.cc:299] Negotiation successful
0808 22:26:15.959010 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":146,"threads_started":1}
I20260808 22:26:15.961149 10215 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d062e632fd74969ac83283ec3a91a56: Neither blocks nor log segments found. Creating new log.
I20260808 22:26:15.961967 10215 log.cc:824] T 00000000000000000000000000000000 P 1d062e632fd74969ac83283ec3a91a56: Log is configured to *not* fsync() on all Append() calls
I20260808 22:26:15.962457 9923 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 22:26:15.963963 10215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d062e632fd74969ac83283ec3a91a56: No bootstrap required, opened a new log
I20260808 22:26:15.967669 10215 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d062e632fd74969ac83283ec3a91a56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d062e632fd74969ac83283ec3a91a56" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 34397 } }
I20260808 22:26:15.967895 10215 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d062e632fd74969ac83283ec3a91a56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:26:15.967972 10215 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d062e632fd74969ac83283ec3a91a56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d062e632fd74969ac83283ec3a91a56, State: Initialized, Role: FOLLOWER
I20260808 22:26:15.968102 10219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:15.964419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59655 (local address 127.9.176.254:34397)
0808 22:26:15.964510 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0808 22:26:15.964513 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:26:15.964763 (+ 250us) server_negotiation.cc:407] Connection header received
0808 22:26:15.964870 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:26:15.964874 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:26:15.964938 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:26:15.965003 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:26:15.965250 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:15.965741 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:15.966522 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:15.966743 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:15.966826 (+ 83us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:26:15.967643 (+ 817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:26:15.967646 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:26:15.967649 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:26:15.967661 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:26:15.967686 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:26:15.967696 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:26:15.967765 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0808 22:26:15.967900 (+ 135us) server_negotiation.cc:299] Negotiation successful
0808 22:26:15.967936 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260808 22:26:15.968437 10215 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d062e632fd74969ac83283ec3a91a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d062e632fd74969ac83283ec3a91a56" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 34397 } }
I20260808 22:26:15.968576 10215 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d062e632fd74969ac83283ec3a91a56 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:26:15.968644 10215 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d062e632fd74969ac83283ec3a91a56 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:26:15.968746 10215 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d062e632fd74969ac83283ec3a91a56 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:26:15.969733 10215 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d062e632fd74969ac83283ec3a91a56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d062e632fd74969ac83283ec3a91a56" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 34397 } }
I20260808 22:26:15.970057 10215 leader_election.cc:302] T 00000000000000000000000000000000 P 1d062e632fd74969ac83283ec3a91a56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d062e632fd74969ac83283ec3a91a56; no voters:
I20260808 22:26:15.970333 10215 leader_election.cc:288] T 00000000000000000000000000000000 P 1d062e632fd74969ac83283ec3a91a56 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:26:15.970410 10230 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d062e632fd74969ac83283ec3a91a56 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:26:15.970618 10230 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d062e632fd74969ac83283ec3a91a56 [term 1 LEADER]: Becoming Leader. State: Replica: 1d062e632fd74969ac83283ec3a91a56, State: Running, Role: LEADER
I20260808 22:26:15.970887 10230 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d062e632fd74969ac83283ec3a91a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d062e632fd74969ac83283ec3a91a56" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 34397 } }
I20260808 22:26:15.971460 10215 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d062e632fd74969ac83283ec3a91a56 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:26:15.971935 10230 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d062e632fd74969ac83283ec3a91a56 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d062e632fd74969ac83283ec3a91a56. Latest consensus state: current_term: 1 leader_uuid: "1d062e632fd74969ac83283ec3a91a56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d062e632fd74969ac83283ec3a91a56" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 34397 } } }
I20260808 22:26:15.972092 10230 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d062e632fd74969ac83283ec3a91a56 [sys.catalog]: This master's current role is: LEADER
I20260808 22:26:15.972505 10238 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:26:15.973276 10233 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d062e632fd74969ac83283ec3a91a56 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d062e632fd74969ac83283ec3a91a56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d062e632fd74969ac83283ec3a91a56" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 34397 } } }
I20260808 22:26:15.973407 10233 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d062e632fd74969ac83283ec3a91a56 [sys.catalog]: This master's current role is: LEADER
I20260808 22:26:15.975351 10238 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:26:15.980602 10238 catalog_manager.cc:1347] Generated new cluster ID: a81d1386b6c246ba965d86e1f4207495
I20260808 22:26:15.980667 10238 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:26:15.993672 10238 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:26:15.994617 10238 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:26:16.004196 10238 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d062e632fd74969ac83283ec3a91a56: Generated new TSK 0
I20260808 22:26:16.005005 10238 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 22:26:16.011792 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 10115
W20260808 22:26:16.022378 10093 connection.cc:535] client connection to 127.9.176.254:34397 recv error: Network error: failed to read from TLS socket (remote: 127.9.176.254:34397): Connection reset by peer (error 104)
2026-08-08T22:26:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1471 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786227706282457-9923-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 22:26:16 dist-test-slave-ncp9 krb5kdc[10262](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 22:26:16 dist-test-slave-ncp9 krb5kdc[10262](info): set up 2 sockets
Aug 08 22:26:16 dist-test-slave-ncp9 krb5kdc[10262](info): commencing operation
krb5kdc: starting...
W20260808 22:26:17.133500 9923 mini_kdc.cc:121] Time spent starting KDC: real 1.097s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786227706282457-9923-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786227706282457-9923-0/krb5kdc/test-user.keytab.
Aug 08 22:26:17 dist-test-slave-ncp9 krb5kdc[10262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227977, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T22:26:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:26:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.176.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.176.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.176.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab.
Entry for principal kudu/127.9.176.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab.
I20260808 22:26:17.249532 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786227706282457-9923-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786227706282457-9923-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786227706282457-9923-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:37373
--webserver_interface=127.9.176.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab
--principal=kudu/127.9.176.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:39645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.176.254:37373
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:26:17.361356 10280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:26:17.361627 10280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:26:17.361668 10280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:26:17.365597 10280 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:26:17.365655 10280 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:26:17.365672 10280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:26:17.365687 10280 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:26:17.365708 10280 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:26:17.370838 10280 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:39645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786227706282457-9923-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786227706282457-9923-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.176.254:37373
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab
--principal=kudu/127.9.176.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:37373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.176.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10280
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786227706282457-9923-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:26:17.371788 10280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:26:17.372720 10280 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:26:17.380980 10292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:26:17.380972 10287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:26:17.380954 10290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:26:17.381196 10280 server_base.cc:1034] running on GCE node
I20260808 22:26:17.381817 10280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:26:17.382853 10280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:26:17.384013 10280 hybrid_clock.cc:648] HybridClock initialized: now 1786227977383995 us; error 29 us; skew 500 ppm
Aug 08 22:26:17 dist-test-slave-ncp9 krb5kdc[10262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227977, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:26:17.386700 10280 init.cc:377] Logged in from keytab as kudu/127.9.176.254@KRBTEST.COM (short username kudu)
I20260808 22:26:17.387533 10280 webserver.cc:460] Webserver started at http://127.9.176.254:35331/ using document root <none> and password file <none>
I20260808 22:26:17.388056 10280 fs_manager.cc:359] Metadata directory not provided
I20260808 22:26:17.388119 10280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:26:17.388314 10280 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:26:17.390331 10280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786227706282457-9923-0/minicluster-data/master-0/data/instance:
uuid: "5d2f06b9c33a4b49b50b04d778c26343"
format_stamp: "Formatted at 2026-08-08 22:26:17 on dist-test-slave-ncp9"
server_key: "ccbd6413150604433f4a3ebe4b6aa82a"
server_key_iv: "9ef5f426cb8e8d39d9b0302bd42a430b"
server_key_version: "encryptionkey@0"
I20260808 22:26:17.390864 10280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786227706282457-9923-0/minicluster-data/master-0/wal/instance:
uuid: "5d2f06b9c33a4b49b50b04d778c26343"
format_stamp: "Formatted at 2026-08-08 22:26:17 on dist-test-slave-ncp9"
server_key: "ccbd6413150604433f4a3ebe4b6aa82a"
server_key_iv: "9ef5f426cb8e8d39d9b0302bd42a430b"
server_key_version: "encryptionkey@0"
I20260808 22:26:17.394380 10280 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 22:26:17.396692 10301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:26:17.397909 10280 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 22:26:17.398029 10280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786227706282457-9923-0/minicluster-data/master-0/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786227706282457-9923-0/minicluster-data/master-0/wal
uuid: "5d2f06b9c33a4b49b50b04d778c26343"
format_stamp: "Formatted at 2026-08-08 22:26:17 on dist-test-slave-ncp9"
server_key: "ccbd6413150604433f4a3ebe4b6aa82a"
server_key_iv: "9ef5f426cb8e8d39d9b0302bd42a430b"
server_key_version: "encryptionkey@0"
I20260808 22:26:17.398131 10280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786227706282457-9923-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786227706282457-9923-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786227706282457-9923-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:26:17.402585 10280 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:26:17.405580 10280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:26:17.405748 10280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:26:17.412339 10280 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.254:37373
I20260808 22:26:17.412379 10376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.254:37373 every 8 connection(s)
I20260808 22:26:17.413246 10280 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
I20260808 22:26:17.415303 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 10280
I20260808 22:26:17.415446 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786227706282457-9923-0/minicluster-data/master-0/wal/instance
I20260808 22:26:17.415757 9923 external_mini_cluster.cc:1428] Setting key e6974e393f2c2e691560149461408200
I20260808 22:26:17.416254 10380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:26:17.420787 10384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:17.417195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37809 (local address 127.9.176.254:37373)
0808 22:26:17.417605 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0808 22:26:17.417622 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 22:26:17.417658 (+ 36us) server_negotiation.cc:407] Connection header received
0808 22:26:17.418286 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:26:17.418301 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:26:17.418566 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:26:17.418839 (+ 273us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:26:17.419646 (+ 807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:26:17.419651 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:26:17.419662 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:26:17.419673 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:26:17.419725 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:26:17.419744 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:26:17.419844 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0808 22:26:17.419951 (+ 107us) server_negotiation.cc:299] Negotiation successful
0808 22:26:17.420081 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":121,"threads_started":1}
I20260808 22:26:17.422533 10380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d2f06b9c33a4b49b50b04d778c26343: Bootstrap starting.
I20260808 22:26:17.423393 9923 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 22:26:17.425107 10380 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d2f06b9c33a4b49b50b04d778c26343: Neither blocks nor log segments found. Creating new log.
I20260808 22:26:17.426079 10380 log.cc:824] T 00000000000000000000000000000000 P 5d2f06b9c33a4b49b50b04d778c26343: Log is configured to *not* fsync() on all Append() calls
I20260808 22:26:17.427419 10384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:17.425607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54113 (local address 127.9.176.254:37373)
0808 22:26:17.425698 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0808 22:26:17.425703 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:26:17.425869 (+ 166us) server_negotiation.cc:407] Connection header received
0808 22:26:17.425949 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:26:17.425951 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:26:17.425992 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:26:17.426070 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:26:17.427090 (+ 1020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:26:17.427093 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:26:17.427096 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:26:17.427107 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:26:17.427134 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:26:17.427147 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:26:17.427233 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0808 22:26:17.427280 (+ 47us) server_negotiation.cc:299] Negotiation successful
0808 22:26:17.427313 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260808 22:26:17.428073 10380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d2f06b9c33a4b49b50b04d778c26343: No bootstrap required, opened a new log
I20260808 22:26:17.431440 10380 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d2f06b9c33a4b49b50b04d778c26343 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d2f06b9c33a4b49b50b04d778c26343" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 37373 } }
I20260808 22:26:17.431659 10380 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5d2f06b9c33a4b49b50b04d778c26343 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:26:17.431706 10380 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d2f06b9c33a4b49b50b04d778c26343 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d2f06b9c33a4b49b50b04d778c26343, State: Initialized, Role: FOLLOWER
I20260808 22:26:17.432053 10380 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d2f06b9c33a4b49b50b04d778c26343 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d2f06b9c33a4b49b50b04d778c26343" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 37373 } }
I20260808 22:26:17.432156 10380 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5d2f06b9c33a4b49b50b04d778c26343 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:26:17.432199 10380 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d2f06b9c33a4b49b50b04d778c26343 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:26:17.432266 10380 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d2f06b9c33a4b49b50b04d778c26343 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:26:17.433216 10380 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d2f06b9c33a4b49b50b04d778c26343 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d2f06b9c33a4b49b50b04d778c26343" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 37373 } }
I20260808 22:26:17.433512 10380 leader_election.cc:302] T 00000000000000000000000000000000 P 5d2f06b9c33a4b49b50b04d778c26343 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d2f06b9c33a4b49b50b04d778c26343; no voters:
I20260808 22:26:17.433770 10380 leader_election.cc:288] T 00000000000000000000000000000000 P 5d2f06b9c33a4b49b50b04d778c26343 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:26:17.433857 10397 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d2f06b9c33a4b49b50b04d778c26343 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:26:17.434074 10397 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d2f06b9c33a4b49b50b04d778c26343 [term 1 LEADER]: Becoming Leader. State: Replica: 5d2f06b9c33a4b49b50b04d778c26343, State: Running, Role: LEADER
I20260808 22:26:17.434386 10397 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d2f06b9c33a4b49b50b04d778c26343 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d2f06b9c33a4b49b50b04d778c26343" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 37373 } }
I20260808 22:26:17.434855 10380 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d2f06b9c33a4b49b50b04d778c26343 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:26:17.435828 10400 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d2f06b9c33a4b49b50b04d778c26343 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d2f06b9c33a4b49b50b04d778c26343. Latest consensus state: current_term: 1 leader_uuid: "5d2f06b9c33a4b49b50b04d778c26343" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d2f06b9c33a4b49b50b04d778c26343" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 37373 } } }
I20260808 22:26:17.435941 10400 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d2f06b9c33a4b49b50b04d778c26343 [sys.catalog]: This master's current role is: LEADER
I20260808 22:26:17.436221 10398 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d2f06b9c33a4b49b50b04d778c26343 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5d2f06b9c33a4b49b50b04d778c26343" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d2f06b9c33a4b49b50b04d778c26343" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 37373 } } }
I20260808 22:26:17.436305 10398 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d2f06b9c33a4b49b50b04d778c26343 [sys.catalog]: This master's current role is: LEADER
I20260808 22:26:17.436316 10411 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:26:17.439038 10411 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:26:17.444661 10411 catalog_manager.cc:1347] Generated new cluster ID: 7f800d7fef194169ae7e58a4753474a4
I20260808 22:26:17.444763 10411 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:26:17.462285 10411 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:26:17.462733 10411 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:26:17.473745 10411 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5d2f06b9c33a4b49b50b04d778c26343: Generated new TSK 0
I20260808 22:26:17.474454 10411 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 22:26:17.485991 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 10280
2026-08-08T22:26:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1474 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786227706282457-9923-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 22:26:17 dist-test-slave-ncp9 krb5kdc[10427](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 22:26:17 dist-test-slave-ncp9 krb5kdc[10427](info): set up 2 sockets
Aug 08 22:26:17 dist-test-slave-ncp9 krb5kdc[10427](info): commencing operation
krb5kdc: starting...
W20260808 22:26:18.614248 9923 mini_kdc.cc:121] Time spent starting KDC: real 1.104s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786227706282457-9923-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786227706282457-9923-0/krb5kdc/test-user.keytab.
Aug 08 22:26:18 dist-test-slave-ncp9 krb5kdc[10427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227978, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T22:26:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:26:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.176.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.176.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.176.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab.
Entry for principal kudu/127.9.176.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab.
I20260808 22:26:18.729583 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786227706282457-9923-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786227706282457-9923-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786227706282457-9923-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:44745
--webserver_interface=127.9.176.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab
--principal=kudu/127.9.176.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:43689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.176.254:44745
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:26:18.841113 10444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:26:18.841355 10444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:26:18.841400 10444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:26:18.845376 10444 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:26:18.845425 10444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:26:18.845443 10444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:26:18.845471 10444 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:26:18.845498 10444 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:26:18.850668 10444 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:43689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786227706282457-9923-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786227706282457-9923-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.176.254:44745
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab
--principal=kudu/127.9.176.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:44745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.176.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10444
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786227706282457-9923-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:26:18.851665 10444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:26:18.852654 10444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:26:18.860817 10452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:26:18.860837 10453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:26:18.860934 10444 server_base.cc:1034] running on GCE node
W20260808 22:26:18.860817 10455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:26:18.861694 10444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:26:18.862741 10444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:26:18.863901 10444 hybrid_clock.cc:648] HybridClock initialized: now 1786227978863887 us; error 24 us; skew 500 ppm
Aug 08 22:26:18 dist-test-slave-ncp9 krb5kdc[10427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227978, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:26:18.870893 10444 init.cc:377] Logged in from keytab as kudu/127.9.176.254@KRBTEST.COM (short username kudu)
I20260808 22:26:18.871781 10444 webserver.cc:460] Webserver started at http://127.9.176.254:37093/ using document root <none> and password file <none>
I20260808 22:26:18.872295 10444 fs_manager.cc:359] Metadata directory not provided
I20260808 22:26:18.872356 10444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:26:18.872563 10444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:26:18.874619 10444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786227706282457-9923-0/minicluster-data/master-0/data/instance:
uuid: "94784c33c6dd45e0b579f38b1d34eb55"
format_stamp: "Formatted at 2026-08-08 22:26:18 on dist-test-slave-ncp9"
server_key: "8c87cba3b38c5016894d439dabbca4f0"
server_key_iv: "1378df03d8a8f1bd1b06735555acf911"
server_key_version: "encryptionkey@0"
I20260808 22:26:18.875154 10444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786227706282457-9923-0/minicluster-data/master-0/wal/instance:
uuid: "94784c33c6dd45e0b579f38b1d34eb55"
format_stamp: "Formatted at 2026-08-08 22:26:18 on dist-test-slave-ncp9"
server_key: "8c87cba3b38c5016894d439dabbca4f0"
server_key_iv: "1378df03d8a8f1bd1b06735555acf911"
server_key_version: "encryptionkey@0"
I20260808 22:26:18.878712 10444 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260808 22:26:18.880973 10464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:26:18.882117 10444 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 22:26:18.882244 10444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786227706282457-9923-0/minicluster-data/master-0/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786227706282457-9923-0/minicluster-data/master-0/wal
uuid: "94784c33c6dd45e0b579f38b1d34eb55"
format_stamp: "Formatted at 2026-08-08 22:26:18 on dist-test-slave-ncp9"
server_key: "8c87cba3b38c5016894d439dabbca4f0"
server_key_iv: "1378df03d8a8f1bd1b06735555acf911"
server_key_version: "encryptionkey@0"
I20260808 22:26:18.882334 10444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786227706282457-9923-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786227706282457-9923-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786227706282457-9923-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:26:18.900197 10444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:26:18.902949 10444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:26:18.903123 10444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:26:18.909947 10444 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.254:44745
I20260808 22:26:18.909961 10538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.254:44745 every 8 connection(s)
I20260808 22:26:18.910871 10444 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
I20260808 22:26:18.913570 10539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:26:18.915549 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 10444
I20260808 22:26:18.915671 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786227706282457-9923-0/minicluster-data/master-0/wal/instance
I20260808 22:26:18.915925 9923 external_mini_cluster.cc:1428] Setting key a6ade18999a67a3ca36769b781968eda
I20260808 22:26:18.920439 10539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94784c33c6dd45e0b579f38b1d34eb55: Bootstrap starting.
I20260808 22:26:18.923043 10539 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 94784c33c6dd45e0b579f38b1d34eb55: Neither blocks nor log segments found. Creating new log.
I20260808 22:26:18.923887 10539 log.cc:824] T 00000000000000000000000000000000 P 94784c33c6dd45e0b579f38b1d34eb55: Log is configured to *not* fsync() on all Append() calls
Aug 08 22:26:18 dist-test-slave-ncp9 krb5kdc[10427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227978, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
I20260808 22:26:18.926072 10539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94784c33c6dd45e0b579f38b1d34eb55: No bootstrap required, opened a new log
I20260808 22:26:18.929955 10539 raft_consensus.cc:348] T 00000000000000000000000000000000 P 94784c33c6dd45e0b579f38b1d34eb55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94784c33c6dd45e0b579f38b1d34eb55" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 44745 } }
I20260808 22:26:18.930171 10539 raft_consensus.cc:374] T 00000000000000000000000000000000 P 94784c33c6dd45e0b579f38b1d34eb55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:26:18.930207 10539 raft_consensus.cc:729] T 00000000000000000000000000000000 P 94784c33c6dd45e0b579f38b1d34eb55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94784c33c6dd45e0b579f38b1d34eb55, State: Initialized, Role: FOLLOWER
I20260808 22:26:18.930400 10545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:18.917333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45259 (local address 127.9.176.254:44745)
0808 22:26:18.917749 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0808 22:26:18.917768 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0808 22:26:18.917810 (+ 42us) server_negotiation.cc:407] Connection header received
0808 22:26:18.918547 (+ 737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:26:18.918573 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:26:18.919016 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:26:18.919382 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:26:18.920038 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:18.921499 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:18.922376 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:18.923062 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:18.923220 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:26:18.925949 (+ 2729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:26:18.925955 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:26:18.925968 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:26:18.925993 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:26:18.927519 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:26:18.928040 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:26:18.928045 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:26:18.928049 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:26:18.928162 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:26:18.928438 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:26:18.928441 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:26:18.928442 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:26:18.928875 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:26:18.929069 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0808 22:26:18.929345 (+ 276us) server_negotiation.cc:299] Negotiation successful
0808 22:26:18.929632 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":105,"threads_started":1}
I20260808 22:26:18.930670 10539 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94784c33c6dd45e0b579f38b1d34eb55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94784c33c6dd45e0b579f38b1d34eb55" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 44745 } }
I20260808 22:26:18.930805 10539 raft_consensus.cc:388] T 00000000000000000000000000000000 P 94784c33c6dd45e0b579f38b1d34eb55 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:26:18.930871 10539 raft_consensus.cc:482] T 00000000000000000000000000000000 P 94784c33c6dd45e0b579f38b1d34eb55 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:26:18.930967 10539 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 94784c33c6dd45e0b579f38b1d34eb55 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:26:18.931778 10539 raft_consensus.cc:504] T 00000000000000000000000000000000 P 94784c33c6dd45e0b579f38b1d34eb55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94784c33c6dd45e0b579f38b1d34eb55" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 44745 } }
I20260808 22:26:18.932103 10539 leader_election.cc:302] T 00000000000000000000000000000000 P 94784c33c6dd45e0b579f38b1d34eb55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 94784c33c6dd45e0b579f38b1d34eb55; no voters:
I20260808 22:26:18.932416 10539 leader_election.cc:288] T 00000000000000000000000000000000 P 94784c33c6dd45e0b579f38b1d34eb55 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:26:18.932487 10548 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 94784c33c6dd45e0b579f38b1d34eb55 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:26:18.932741 10548 raft_consensus.cc:686] T 00000000000000000000000000000000 P 94784c33c6dd45e0b579f38b1d34eb55 [term 1 LEADER]: Becoming Leader. State: Replica: 94784c33c6dd45e0b579f38b1d34eb55, State: Running, Role: LEADER
I20260808 22:26:18.933221 9923 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 22:26:18.933655 10539 sys_catalog.cc:564] T 00000000000000000000000000000000 P 94784c33c6dd45e0b579f38b1d34eb55 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:26:18.933126 10548 consensus_queue.cc:237] T 00000000000000000000000000000000 P 94784c33c6dd45e0b579f38b1d34eb55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94784c33c6dd45e0b579f38b1d34eb55" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 44745 } }
I20260808 22:26:18.936915 10549 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94784c33c6dd45e0b579f38b1d34eb55 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "94784c33c6dd45e0b579f38b1d34eb55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94784c33c6dd45e0b579f38b1d34eb55" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 44745 } } }
I20260808 22:26:18.936962 10550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94784c33c6dd45e0b579f38b1d34eb55 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94784c33c6dd45e0b579f38b1d34eb55. Latest consensus state: current_term: 1 leader_uuid: "94784c33c6dd45e0b579f38b1d34eb55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94784c33c6dd45e0b579f38b1d34eb55" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 44745 } } }
I20260808 22:26:18.937050 10549 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94784c33c6dd45e0b579f38b1d34eb55 [sys.catalog]: This master's current role is: LEADER
I20260808 22:26:18.937071 10550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94784c33c6dd45e0b579f38b1d34eb55 [sys.catalog]: This master's current role is: LEADER
I20260808 22:26:18.937487 10569 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:26:18.940551 10569 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:26:18.943464 10545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:18.935418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41993 (local address 127.9.176.254:44745)
0808 22:26:18.935518 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0808 22:26:18.935522 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:26:18.935534 (+ 12us) server_negotiation.cc:407] Connection header received
0808 22:26:18.935578 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:26:18.935581 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:26:18.935620 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:26:18.935687 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:26:18.936466 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:18.937296 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:18.938202 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:18.938489 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:18.938589 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:26:18.939989 (+ 1400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:26:18.939993 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:26:18.939995 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:26:18.940019 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:26:18.941686 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:26:18.942196 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:26:18.942199 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:26:18.942201 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:26:18.942290 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:26:18.942594 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:26:18.942597 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:26:18.942599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:26:18.942815 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:26:18.942945 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0808 22:26:18.943198 (+ 253us) server_negotiation.cc:299] Negotiation successful
0808 22:26:18.943312 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260808 22:26:18.946192 10569 catalog_manager.cc:1347] Generated new cluster ID: a91fe62b0ed9496381fd7f2ce5ae539f
I20260808 22:26:18.946256 10569 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:26:18.957221 10569 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:26:18.958087 10569 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:26:18.962399 10569 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 94784c33c6dd45e0b579f38b1d34eb55: Generated new TSK 0
I20260808 22:26:18.962986 10569 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 22:26:18.971002 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 10444
2026-08-08T22:26:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1491 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786227706282457-9923-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 22:26:19 dist-test-slave-ncp9 krb5kdc[10581](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 22:26:19 dist-test-slave-ncp9 krb5kdc[10581](info): set up 2 sockets
Aug 08 22:26:19 dist-test-slave-ncp9 krb5kdc[10581](info): commencing operation
krb5kdc: starting...
W20260808 22:26:20.096589 9923 mini_kdc.cc:121] Time spent starting KDC: real 1.095s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786227706282457-9923-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786227706282457-9923-0/krb5kdc/test-user.keytab.
Aug 08 22:26:20 dist-test-slave-ncp9 krb5kdc[10581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227980, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T22:26:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:26:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.176.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.176.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.176.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab.
Entry for principal kudu/127.9.176.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab.
I20260808 22:26:20.215436 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786227706282457-9923-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786227706282457-9923-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786227706282457-9923-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:42151
--webserver_interface=127.9.176.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab
--principal=kudu/127.9.176.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:39063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.176.254:42151
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:26:20.322993 10598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:26:20.323225 10598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:26:20.323262 10598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:26:20.327127 10598 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:26:20.327178 10598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:26:20.327195 10598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:26:20.327209 10598 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:26:20.327222 10598 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:26:20.332273 10598 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:39063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786227706282457-9923-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786227706282457-9923-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.176.254:42151
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab
--principal=kudu/127.9.176.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:42151
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.176.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786227706282457-9923-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:26:20.333204 10598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:26:20.334127 10598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:26:20.342068 10608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:26:20.342079 10611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:26:20.342108 10609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:26:20.342330 10598 server_base.cc:1034] running on GCE node
I20260808 22:26:20.342898 10598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:26:20.343902 10598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:26:20.345043 10598 hybrid_clock.cc:648] HybridClock initialized: now 1786227980345026 us; error 38 us; skew 500 ppm
Aug 08 22:26:20 dist-test-slave-ncp9 krb5kdc[10581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227980, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:26:20.347769 10598 init.cc:377] Logged in from keytab as kudu/127.9.176.254@KRBTEST.COM (short username kudu)
I20260808 22:26:20.348630 10598 webserver.cc:460] Webserver started at http://127.9.176.254:33791/ using document root <none> and password file <none>
I20260808 22:26:20.349188 10598 fs_manager.cc:359] Metadata directory not provided
I20260808 22:26:20.349255 10598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:26:20.349462 10598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:26:20.351477 10598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786227706282457-9923-0/minicluster-data/master-0/data/instance:
uuid: "b262769b8c224629b10dd4481c06d9fd"
format_stamp: "Formatted at 2026-08-08 22:26:20 on dist-test-slave-ncp9"
server_key: "62d5e867baa44a606c4b68decf3ff52e"
server_key_iv: "728871c6b1c77fc03f393967e573acfc"
server_key_version: "encryptionkey@0"
I20260808 22:26:20.352022 10598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786227706282457-9923-0/minicluster-data/master-0/wal/instance:
uuid: "b262769b8c224629b10dd4481c06d9fd"
format_stamp: "Formatted at 2026-08-08 22:26:20 on dist-test-slave-ncp9"
server_key: "62d5e867baa44a606c4b68decf3ff52e"
server_key_iv: "728871c6b1c77fc03f393967e573acfc"
server_key_version: "encryptionkey@0"
I20260808 22:26:20.355716 10598 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260808 22:26:20.358223 10623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:26:20.359460 10598 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 22:26:20.359588 10598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786227706282457-9923-0/minicluster-data/master-0/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786227706282457-9923-0/minicluster-data/master-0/wal
uuid: "b262769b8c224629b10dd4481c06d9fd"
format_stamp: "Formatted at 2026-08-08 22:26:20 on dist-test-slave-ncp9"
server_key: "62d5e867baa44a606c4b68decf3ff52e"
server_key_iv: "728871c6b1c77fc03f393967e573acfc"
server_key_version: "encryptionkey@0"
I20260808 22:26:20.359679 10598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786227706282457-9923-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786227706282457-9923-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786227706282457-9923-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:26:20.392997 10598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:26:20.396116 10598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:26:20.396301 10598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:26:20.403524 10598 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.254:42151
I20260808 22:26:20.403539 10705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.254:42151 every 8 connection(s)
I20260808 22:26:20.404448 10598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
I20260808 22:26:20.407151 10707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:26:20.411445 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 10598
I20260808 22:26:20.411592 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786227706282457-9923-0/minicluster-data/master-0/wal/instance
I20260808 22:26:20.411887 9923 external_mini_cluster.cc:1428] Setting key 48ffc24d908e604a466142f4e515df04
I20260808 22:26:20.413061 10707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b262769b8c224629b10dd4481c06d9fd: Bootstrap starting.
I20260808 22:26:20.415413 10707 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b262769b8c224629b10dd4481c06d9fd: Neither blocks nor log segments found. Creating new log.
I20260808 22:26:20.416121 10707 log.cc:824] T 00000000000000000000000000000000 P b262769b8c224629b10dd4481c06d9fd: Log is configured to *not* fsync() on all Append() calls
I20260808 22:26:20.418110 10707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b262769b8c224629b10dd4481c06d9fd: No bootstrap required, opened a new log
I20260808 22:26:20.420476 10712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:20.413249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39575 (local address 127.9.176.254:42151)
0808 22:26:20.413643 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0808 22:26:20.413660 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 22:26:20.413693 (+ 33us) server_negotiation.cc:407] Connection header received
0808 22:26:20.414353 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:26:20.414370 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:26:20.414703 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:26:20.414974 (+ 271us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:26:20.415286 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:20.416296 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:20.417396 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:20.417860 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:20.417983 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:26:20.418810 (+ 827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:26:20.418815 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:26:20.418829 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:26:20.419245 (+ 416us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:26:20.419308 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:26:20.419332 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:26:20.419408 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 22:26:20.419560 (+ 152us) server_negotiation.cc:299] Negotiation successful
0808 22:26:20.419756 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":109,"threads_started":1}
I20260808 22:26:20.421802 10707 raft_consensus.cc:348] T 00000000000000000000000000000000 P b262769b8c224629b10dd4481c06d9fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b262769b8c224629b10dd4481c06d9fd" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 42151 } }
I20260808 22:26:20.422014 10707 raft_consensus.cc:374] T 00000000000000000000000000000000 P b262769b8c224629b10dd4481c06d9fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:26:20.422075 10707 raft_consensus.cc:729] T 00000000000000000000000000000000 P b262769b8c224629b10dd4481c06d9fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b262769b8c224629b10dd4481c06d9fd, State: Initialized, Role: FOLLOWER
I20260808 22:26:20.422487 10707 consensus_queue.cc:260] T 00000000000000000000000000000000 P b262769b8c224629b10dd4481c06d9fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b262769b8c224629b10dd4481c06d9fd" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 42151 } }
I20260808 22:26:20.422654 10707 raft_consensus.cc:388] T 00000000000000000000000000000000 P b262769b8c224629b10dd4481c06d9fd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:26:20.422724 10707 raft_consensus.cc:482] T 00000000000000000000000000000000 P b262769b8c224629b10dd4481c06d9fd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:26:20.422821 10707 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b262769b8c224629b10dd4481c06d9fd [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:26:20.423321 9923 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 22:26:20.423846 10707 raft_consensus.cc:504] T 00000000000000000000000000000000 P b262769b8c224629b10dd4481c06d9fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b262769b8c224629b10dd4481c06d9fd" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 42151 } }
I20260808 22:26:20.424206 10707 leader_election.cc:302] T 00000000000000000000000000000000 P b262769b8c224629b10dd4481c06d9fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b262769b8c224629b10dd4481c06d9fd; no voters:
I20260808 22:26:20.424568 10707 leader_election.cc:288] T 00000000000000000000000000000000 P b262769b8c224629b10dd4481c06d9fd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:26:20.424652 10716 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b262769b8c224629b10dd4481c06d9fd [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:26:20.424878 10716 raft_consensus.cc:686] T 00000000000000000000000000000000 P b262769b8c224629b10dd4481c06d9fd [term 1 LEADER]: Becoming Leader. State: Replica: b262769b8c224629b10dd4481c06d9fd, State: Running, Role: LEADER
I20260808 22:26:20.425343 10716 consensus_queue.cc:237] T 00000000000000000000000000000000 P b262769b8c224629b10dd4481c06d9fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b262769b8c224629b10dd4481c06d9fd" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 42151 } }
I20260808 22:26:20.425908 10707 sys_catalog.cc:564] T 00000000000000000000000000000000 P b262769b8c224629b10dd4481c06d9fd [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:26:20.427238 10723 sys_catalog.cc:455] T 00000000000000000000000000000000 P b262769b8c224629b10dd4481c06d9fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader b262769b8c224629b10dd4481c06d9fd. Latest consensus state: current_term: 1 leader_uuid: "b262769b8c224629b10dd4481c06d9fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b262769b8c224629b10dd4481c06d9fd" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 42151 } } }
I20260808 22:26:20.427377 10723 sys_catalog.cc:458] T 00000000000000000000000000000000 P b262769b8c224629b10dd4481c06d9fd [sys.catalog]: This master's current role is: LEADER
I20260808 22:26:20.428359 10724 sys_catalog.cc:455] T 00000000000000000000000000000000 P b262769b8c224629b10dd4481c06d9fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b262769b8c224629b10dd4481c06d9fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b262769b8c224629b10dd4481c06d9fd" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 42151 } } }
I20260808 22:26:20.428452 10724 sys_catalog.cc:458] T 00000000000000000000000000000000 P b262769b8c224629b10dd4481c06d9fd [sys.catalog]: This master's current role is: LEADER
I20260808 22:26:20.430110 10712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:20.425495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39289 (local address 127.9.176.254:42151)
0808 22:26:20.425586 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0808 22:26:20.425590 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:26:20.425772 (+ 182us) server_negotiation.cc:407] Connection header received
0808 22:26:20.425869 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:26:20.425871 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:26:20.425915 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:26:20.425994 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:26:20.426272 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:20.427034 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:20.428129 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:20.428440 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:20.428538 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:26:20.429506 (+ 968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:26:20.429510 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:26:20.429513 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:26:20.429525 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:26:20.429554 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:26:20.429568 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:26:20.429628 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0808 22:26:20.429926 (+ 298us) server_negotiation.cc:299] Negotiation successful
0808 22:26:20.429977 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
W20260808 22:26:20.430729 10744 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b262769b8c224629b10dd4481c06d9fd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 22:26:20.430805 10744 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 22:26:20.430869 10739 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:26:20.431612 10739 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:26:20.436575 10739 catalog_manager.cc:1347] Generated new cluster ID: cf5ff1ed0a7146fa831783c00e6da5fa
I20260808 22:26:20.436640 10739 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:26:20.449592 10739 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:26:20.450783 10739 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:26:20.461030 10739 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b262769b8c224629b10dd4481c06d9fd: Generated new TSK 0
I20260808 22:26:20.461818 10739 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 22:26:20.468498 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 10598
2026-08-08T22:26:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1492 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786227706282457-9923-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 22:26:20 dist-test-slave-ncp9 krb5kdc[10759](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 22:26:20 dist-test-slave-ncp9 krb5kdc[10759](info): set up 2 sockets
Aug 08 22:26:20 dist-test-slave-ncp9 krb5kdc[10759](info): commencing operation
krb5kdc: starting...
W20260808 22:26:21.594696 9923 mini_kdc.cc:121] Time spent starting KDC: real 1.101s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786227706282457-9923-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786227706282457-9923-0/krb5kdc/test-user.keytab.
Aug 08 22:26:21 dist-test-slave-ncp9 krb5kdc[10759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T22:26:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:26:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.176.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.176.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.176.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab.
Entry for principal kudu/127.9.176.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab.
I20260808 22:26:21.714238 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786227706282457-9923-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786227706282457-9923-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786227706282457-9923-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:40221
--webserver_interface=127.9.176.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab
--principal=kudu/127.9.176.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:33595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.176.254:40221
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:26:21.827724 10776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:26:21.828023 10776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:26:21.828088 10776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:26:21.832082 10776 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:26:21.832159 10776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:26:21.832192 10776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:26:21.832218 10776 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:26:21.832253 10776 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:26:21.837602 10776 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:33595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786227706282457-9923-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786227706282457-9923-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.176.254:40221
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab
--principal=kudu/127.9.176.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:40221
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.176.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10776
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786227706282457-9923-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:26:21.838681 10776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:26:21.839731 10776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:26:21.848935 10786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:26:21.849081 10776 server_base.cc:1034] running on GCE node
W20260808 22:26:21.848960 10792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:26:21.848992 10785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:26:21.849730 10776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:26:21.850833 10776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:26:21.851984 10776 hybrid_clock.cc:648] HybridClock initialized: now 1786227981851964 us; error 37 us; skew 500 ppm
Aug 08 22:26:21 dist-test-slave-ncp9 krb5kdc[10759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227981, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:26:21.854727 10776 init.cc:377] Logged in from keytab as kudu/127.9.176.254@KRBTEST.COM (short username kudu)
I20260808 22:26:21.855571 10776 webserver.cc:460] Webserver started at http://127.9.176.254:44565/ using document root <none> and password file <none>
I20260808 22:26:21.856110 10776 fs_manager.cc:359] Metadata directory not provided
I20260808 22:26:21.856177 10776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:26:21.856375 10776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:26:21.858393 10776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786227706282457-9923-0/minicluster-data/master-0/data/instance:
uuid: "99ebfde2a5724c89b67baf648625fe32"
format_stamp: "Formatted at 2026-08-08 22:26:21 on dist-test-slave-ncp9"
server_key: "5db2cca6bc64911005f6ee2775ad6896"
server_key_iv: "342c4fd406ce00152e6011f755bab77e"
server_key_version: "encryptionkey@0"
I20260808 22:26:21.859000 10776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786227706282457-9923-0/minicluster-data/master-0/wal/instance:
uuid: "99ebfde2a5724c89b67baf648625fe32"
format_stamp: "Formatted at 2026-08-08 22:26:21 on dist-test-slave-ncp9"
server_key: "5db2cca6bc64911005f6ee2775ad6896"
server_key_iv: "342c4fd406ce00152e6011f755bab77e"
server_key_version: "encryptionkey@0"
I20260808 22:26:21.862645 10776 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 22:26:21.864974 10801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:26:21.866148 10776 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 22:26:21.866261 10776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786227706282457-9923-0/minicluster-data/master-0/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786227706282457-9923-0/minicluster-data/master-0/wal
uuid: "99ebfde2a5724c89b67baf648625fe32"
format_stamp: "Formatted at 2026-08-08 22:26:21 on dist-test-slave-ncp9"
server_key: "5db2cca6bc64911005f6ee2775ad6896"
server_key_iv: "342c4fd406ce00152e6011f755bab77e"
server_key_version: "encryptionkey@0"
I20260808 22:26:21.866353 10776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786227706282457-9923-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786227706282457-9923-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786227706282457-9923-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:26:21.871165 10776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:26:21.874096 10776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:26:21.874277 10776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:26:21.881490 10776 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.254:40221
I20260808 22:26:21.881531 10871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.254:40221 every 8 connection(s)
I20260808 22:26:21.882444 10776 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
I20260808 22:26:21.885193 10873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:26:21.890651 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 10776
I20260808 22:26:21.890789 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786227706282457-9923-0/minicluster-data/master-0/wal/instance
I20260808 22:26:21.890786 10873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99ebfde2a5724c89b67baf648625fe32: Bootstrap starting.
I20260808 22:26:21.891064 9923 external_mini_cluster.cc:1428] Setting key 7798e68c964ebb3a2fdcc40d5f8742bc
I20260808 22:26:21.893281 10873 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 99ebfde2a5724c89b67baf648625fe32: Neither blocks nor log segments found. Creating new log.
I20260808 22:26:21.894017 10873 log.cc:824] T 00000000000000000000000000000000 P 99ebfde2a5724c89b67baf648625fe32: Log is configured to *not* fsync() on all Append() calls
I20260808 22:26:21.896245 10873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99ebfde2a5724c89b67baf648625fe32: No bootstrap required, opened a new log
I20260808 22:26:21.896296 10876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:21.892359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34047 (local address 127.9.176.254:40221)
0808 22:26:21.892770 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0808 22:26:21.892784 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 22:26:21.892819 (+ 35us) server_negotiation.cc:407] Connection header received
0808 22:26:21.893514 (+ 695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:26:21.893541 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:26:21.893821 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:26:21.894093 (+ 272us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:26:21.894928 (+ 835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:26:21.894934 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:26:21.894949 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:26:21.894961 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:26:21.895033 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:26:21.895061 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:26:21.895164 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0808 22:26:21.895283 (+ 119us) server_negotiation.cc:299] Negotiation successful
0808 22:26:21.895427 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":116,"threads_started":1}
I20260808 22:26:21.898900 9923 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 22:26:21.900103 10873 raft_consensus.cc:348] T 00000000000000000000000000000000 P 99ebfde2a5724c89b67baf648625fe32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99ebfde2a5724c89b67baf648625fe32" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 40221 } }
I20260808 22:26:21.900389 10873 raft_consensus.cc:374] T 00000000000000000000000000000000 P 99ebfde2a5724c89b67baf648625fe32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:26:21.900453 10873 raft_consensus.cc:729] T 00000000000000000000000000000000 P 99ebfde2a5724c89b67baf648625fe32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ebfde2a5724c89b67baf648625fe32, State: Initialized, Role: FOLLOWER
I20260808 22:26:21.900933 10873 consensus_queue.cc:260] T 00000000000000000000000000000000 P 99ebfde2a5724c89b67baf648625fe32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99ebfde2a5724c89b67baf648625fe32" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 40221 } }
I20260808 22:26:21.901098 10873 raft_consensus.cc:388] T 00000000000000000000000000000000 P 99ebfde2a5724c89b67baf648625fe32 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:26:21.901171 10873 raft_consensus.cc:482] T 00000000000000000000000000000000 P 99ebfde2a5724c89b67baf648625fe32 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:26:21.901275 10873 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 99ebfde2a5724c89b67baf648625fe32 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:26:21.902236 10873 raft_consensus.cc:504] T 00000000000000000000000000000000 P 99ebfde2a5724c89b67baf648625fe32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99ebfde2a5724c89b67baf648625fe32" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 40221 } }
I20260808 22:26:21.902591 10873 leader_election.cc:302] T 00000000000000000000000000000000 P 99ebfde2a5724c89b67baf648625fe32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 99ebfde2a5724c89b67baf648625fe32; no voters:
I20260808 22:26:21.902697 10876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:21.901147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36413 (local address 127.9.176.254:40221)
0808 22:26:21.901249 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0808 22:26:21.901253 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:26:21.901447 (+ 194us) server_negotiation.cc:407] Connection header received
0808 22:26:21.901564 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:26:21.901567 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:26:21.901600 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:26:21.901676 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:26:21.902319 (+ 643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:26:21.902322 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:26:21.902324 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:26:21.902336 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:26:21.902361 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:26:21.902373 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:26:21.902447 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0808 22:26:21.902515 (+ 68us) server_negotiation.cc:299] Negotiation successful
0808 22:26:21.902557 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260808 22:26:21.902944 10873 leader_election.cc:288] T 00000000000000000000000000000000 P 99ebfde2a5724c89b67baf648625fe32 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:26:21.903023 10887 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 99ebfde2a5724c89b67baf648625fe32 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:26:21.903221 10887 raft_consensus.cc:686] T 00000000000000000000000000000000 P 99ebfde2a5724c89b67baf648625fe32 [term 1 LEADER]: Becoming Leader. State: Replica: 99ebfde2a5724c89b67baf648625fe32, State: Running, Role: LEADER
I20260808 22:26:21.903508 10887 consensus_queue.cc:237] T 00000000000000000000000000000000 P 99ebfde2a5724c89b67baf648625fe32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99ebfde2a5724c89b67baf648625fe32" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 40221 } }
I20260808 22:26:21.904120 10873 sys_catalog.cc:564] T 00000000000000000000000000000000 P 99ebfde2a5724c89b67baf648625fe32 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:26:21.904629 10890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99ebfde2a5724c89b67baf648625fe32 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 99ebfde2a5724c89b67baf648625fe32. Latest consensus state: current_term: 1 leader_uuid: "99ebfde2a5724c89b67baf648625fe32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99ebfde2a5724c89b67baf648625fe32" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 40221 } } }
I20260808 22:26:21.904755 10890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99ebfde2a5724c89b67baf648625fe32 [sys.catalog]: This master's current role is: LEADER
I20260808 22:26:21.905088 10888 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99ebfde2a5724c89b67baf648625fe32 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "99ebfde2a5724c89b67baf648625fe32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99ebfde2a5724c89b67baf648625fe32" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 40221 } } }
I20260808 22:26:21.905189 10888 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99ebfde2a5724c89b67baf648625fe32 [sys.catalog]: This master's current role is: LEADER
W20260808 22:26:21.908429 10912 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 99ebfde2a5724c89b67baf648625fe32: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 22:26:21.908494 10912 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 22:26:21.908595 10900 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:26:21.909502 10900 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:26:21.914664 10900 catalog_manager.cc:1347] Generated new cluster ID: b1c0e96f3c5e431bad8bc7978c0a0300
I20260808 22:26:21.914727 10900 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:26:21.931865 10900 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:26:21.932181 10900 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:26:21.938254 10900 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 99ebfde2a5724c89b67baf648625fe32: Generated new TSK 0
I20260808 22:26:21.938881 10900 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 22:26:21.947196 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 10776
2026-08-08T22:26:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1484 ms)
[----------] 18 tests from AuthTokenIssuingTest (26649 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786227706282457-9923-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 22:26:22 dist-test-slave-ncp9 krb5kdc[10921](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 22:26:22 dist-test-slave-ncp9 krb5kdc[10921](info): set up 2 sockets
Aug 08 22:26:22 dist-test-slave-ncp9 krb5kdc[10921](info): commencing operation
krb5kdc: starting...
W20260808 22:26:23.077291 9923 mini_kdc.cc:121] Time spent starting KDC: real 1.099s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786227706282457-9923-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786227706282457-9923-0/krb5kdc/test-user.keytab.
Aug 08 22:26:23 dist-test-slave-ncp9 krb5kdc[10921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T22:26:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:26:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.176.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.176.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.176.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab.
Entry for principal kudu/127.9.176.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab.
I20260808 22:26:23.195293 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786227706282457-9923-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786227706282457-9923-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786227706282457-9923-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:35165
--webserver_interface=127.9.176.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab
--principal=kudu/127.9.176.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:35643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.176.254:35165,127.9.176.253:42637,127.9.176.252:46809
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:26:23.301579 10941 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 22:26:23.301836 10941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:26:23.301867 10941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:26:23.301920 10941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:26:23.305768 10941 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:26:23.305815 10941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:26:23.305841 10941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:26:23.305860 10941 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:26:23.305881 10941 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:26:23.310864 10941 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:35643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786227706282457-9923-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786227706282457-9923-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.176.254:35165,127.9.176.253:42637,127.9.176.252:46809
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab
--principal=kudu/127.9.176.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:35165
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.176.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10941
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786227706282457-9923-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:26:23.311800 10941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:26:23.312754 10941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:26:23.320866 10953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:26:23.320858 10950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:26:23.320974 10941 server_base.cc:1034] running on GCE node
W20260808 22:26:23.320927 10951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:26:23.321768 10941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:26:23.322885 10941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:26:23.324071 10941 hybrid_clock.cc:648] HybridClock initialized: now 1786227983324050 us; error 30 us; skew 500 ppm
Aug 08 22:26:23 dist-test-slave-ncp9 krb5kdc[10921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227983, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:26:23.326753 10941 init.cc:377] Logged in from keytab as kudu/127.9.176.254@KRBTEST.COM (short username kudu)
I20260808 22:26:23.327570 10941 webserver.cc:460] Webserver started at http://127.9.176.254:39083/ using document root <none> and password file <none>
I20260808 22:26:23.328078 10941 fs_manager.cc:359] Metadata directory not provided
I20260808 22:26:23.328145 10941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:26:23.328341 10941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:26:23.330399 10941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786227706282457-9923-0/minicluster-data/master-0/data/instance:
uuid: "f180cbc2339f4ee8abe6d13f52d13268"
format_stamp: "Formatted at 2026-08-08 22:26:23 on dist-test-slave-ncp9"
server_key: "01a6a760dad1bf32a145a14ae999221b"
server_key_iv: "16807a8bc9c7388ef0451a611cd79fe3"
server_key_version: "encryptionkey@0"
I20260808 22:26:23.330935 10941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786227706282457-9923-0/minicluster-data/master-0/wal/instance:
uuid: "f180cbc2339f4ee8abe6d13f52d13268"
format_stamp: "Formatted at 2026-08-08 22:26:23 on dist-test-slave-ncp9"
server_key: "01a6a760dad1bf32a145a14ae999221b"
server_key_iv: "16807a8bc9c7388ef0451a611cd79fe3"
server_key_version: "encryptionkey@0"
I20260808 22:26:23.334419 10941 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 22:26:23.336642 10963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:26:23.337831 10941 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 22:26:23.337945 10941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786227706282457-9923-0/minicluster-data/master-0/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786227706282457-9923-0/minicluster-data/master-0/wal
uuid: "f180cbc2339f4ee8abe6d13f52d13268"
format_stamp: "Formatted at 2026-08-08 22:26:23 on dist-test-slave-ncp9"
server_key: "01a6a760dad1bf32a145a14ae999221b"
server_key_iv: "16807a8bc9c7388ef0451a611cd79fe3"
server_key_version: "encryptionkey@0"
I20260808 22:26:23.338035 10941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786227706282457-9923-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786227706282457-9923-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786227706282457-9923-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:26:23.347998 10941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:26:23.350896 10941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:26:23.351063 10941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:26:23.357488 10941 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.254:35165
I20260808 22:26:23.357739 11041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.254:35165 every 8 connection(s)
I20260808 22:26:23.358371 10941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
I20260808 22:26:23.360338 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 10941
I20260808 22:26:23.360466 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786227706282457-9923-0/minicluster-data/master-0/wal/instance
I20260808 22:26:23.360754 9923 external_mini_cluster.cc:1428] Setting key 2b8c8d4af0fb95188b6f8b60c3b30831
I20260808 22:26:23.361229 11046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:26:23.365187 11046 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.176.254" port: 35165 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.9.176.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.176.253@KRBTEST.COM" created.
Aug 08 22:26:23 dist-test-slave-ncp9 krb5kdc[10921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227983, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.254@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
I20260808 22:26:23.379752 11055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:23.367405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53082 (local address 127.9.176.254:35165)
0808 22:26:23.367806 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0808 22:26:23.367823 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 22:26:23.368050 (+ 227us) server_negotiation.cc:407] Connection header received
0808 22:26:23.369043 (+ 993us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:26:23.369047 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:26:23.369238 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:26:23.369398 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:26:23.370241 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:23.371146 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:23.372165 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:23.372529 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:23.372668 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:26:23.375296 (+ 2628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:26:23.375301 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:26:23.375311 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:26:23.375332 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:26:23.376817 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:26:23.377382 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:26:23.377386 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:26:23.377387 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:26:23.377513 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:26:23.377859 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:26:23.377861 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:26:23.377862 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:26:23.378218 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:26:23.378316 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0808 22:26:23.378773 (+ 457us) server_negotiation.cc:299] Negotiation successful
0808 22:26:23.378986 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":268,"thread_start_us":108,"threads_started":1}
I20260808 22:26:23.379757 11056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:23.367406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:35165 (local address 127.0.0.1:53082)
0808 22:26:23.367918 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0808 22:26:23.367954 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0808 22:26:23.368740 (+ 786us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:26:23.369411 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:26:23.369416 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:26:23.369829 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:26:23.370102 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:26:23.370116 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:26:23.371250 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:26:23.371254 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:26:23.372060 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:26:23.372071 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:26:23.372669 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:26:23.372673 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:26:23.372824 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:26:23.373438 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:26:23.373459 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:26:23.375146 (+ 1687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:26:23.376953 (+ 1807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:26:23.376964 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:26:23.376975 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:26:23.377279 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:26:23.377625 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:26:23.377628 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:26:23.377630 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:26:23.377775 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:26:23.378332 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:26:23.378339 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:26:23.378530 (+ 191us) client_negotiation.cc:769] Sending connection context
0808 22:26:23.378719 (+ 189us) client_negotiation.cc:240] Negotiation successful
0808 22:26:23.379076 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"mutex_wait_us":70,"spinlock_wait_cycles":13184,"thread_start_us":160,"threads_started":1}
I20260808 22:26:23.381929 11046 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.176.253" port: 42637 } has no permanent_uuid. Determining permanent_uuid...
I20260808 22:26:23.382589 11056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:23.382338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.253:42637 (local address 127.0.0.1:59554)
0808 22:26:23.382435 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0808 22:26:23.382513 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.253:42637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260808 22:26:23.382769 10967 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.176.253:42637: connect: Connection refused (error 111)
I20260808 22:26:23.383594 11056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:23.383390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.253:42637 (local address 127.0.0.1:59570)
0808 22:26:23.383486 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0808 22:26:23.383543 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.253:42637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260808 22:26:23.383710 11046 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.176.253:42637: Network error: Client connection negotiation failed: client connection to 127.9.176.253:42637: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.176.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.253.keytab.
Entry for principal kudu/127.9.176.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.253.keytab.
I20260808 22:26:23.387257 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786227706282457-9923-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786227706282457-9923-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786227706282457-9923-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786227706282457-9923-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.253:42637
--webserver_interface=127.9.176.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.253.keytab
--principal=kudu/127.9.176.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:35643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.176.254:35165,127.9.176.253:42637,127.9.176.252:46809
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 22:26:23.432946 11046 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.176.253" port: 42637 } attempt: 1
I20260808 22:26:23.433878 11056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:23.433550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.253:42637 (local address 127.0.0.1:59586)
0808 22:26:23.433712 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0808 22:26:23.433794 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.253:42637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260808 22:26:23.434625 11056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:23.434366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.253:42637 (local address 127.0.0.1:59594)
0808 22:26:23.434513 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0808 22:26:23.434564 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.253:42637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260808 22:26:23.434770 11046 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.176.253:42637: Network error: Client connection negotiation failed: client connection to 127.9.176.253:42637: connect: Connection refused (error 111)
W20260808 22:26:23.500355 11064 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 22:26:23.500639 11064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:26:23.500674 11064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:26:23.500736 11064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260808 22:26:23.502957 11046 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.176.253" port: 42637 } attempt: 2
I20260808 22:26:23.503795 11056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:23.503502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.253:42637 (local address 127.0.0.1:59600)
0808 22:26:23.503652 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0808 22:26:23.503718 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.253:42637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260808 22:26:23.504483 11056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:23.504277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.253:42637 (local address 127.0.0.1:59610)
0808 22:26:23.504381 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0808 22:26:23.504427 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.253:42637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260808 22:26:23.504637 11046 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.176.253:42637: Network error: Client connection negotiation failed: client connection to 127.9.176.253:42637: connect: Connection refused (error 111)
W20260808 22:26:23.505028 11064 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:26:23.505082 11064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:26:23.505118 11064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:26:23.505148 11064 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:26:23.505177 11064 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:26:23.510454 11064 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:35643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786227706282457-9923-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786227706282457-9923-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.9.176.254:35165,127.9.176.253:42637,127.9.176.252:46809
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.253.keytab
--principal=kudu/127.9.176.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.253:42637
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786227706282457-9923-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.9.176.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11064
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786227706282457-9923-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:26:23.511508 11064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:26:23.512516 11064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:26:23.520555 11075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:26:23.520720 11064 server_base.cc:1034] running on GCE node
W20260808 22:26:23.520552 11073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:26:23.520558 11083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:26:23.521385 11064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:26:23.522475 11064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:26:23.523638 11064 hybrid_clock.cc:648] HybridClock initialized: now 1786227983523610 us; error 39 us; skew 500 ppm
Aug 08 22:26:23 dist-test-slave-ncp9 krb5kdc[10921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227983, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:26:23.526360 11064 init.cc:377] Logged in from keytab as kudu/127.9.176.253@KRBTEST.COM (short username kudu)
I20260808 22:26:23.527194 11064 webserver.cc:460] Webserver started at http://127.9.176.253:35347/ using document root <none> and password file <none>
I20260808 22:26:23.527700 11064 fs_manager.cc:359] Metadata directory not provided
I20260808 22:26:23.527787 11064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:26:23.527997 11064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:26:23.530033 11064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786227706282457-9923-0/minicluster-data/master-1/data/instance:
uuid: "81a36fa646864b2080b554566106f120"
format_stamp: "Formatted at 2026-08-08 22:26:23 on dist-test-slave-ncp9"
server_key: "1eecd907cb2ca7fca91e95c9403c1ed0"
server_key_iv: "ddfa298a8acfd834983c3331db05346f"
server_key_version: "encryptionkey@0"
I20260808 22:26:23.530577 11064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786227706282457-9923-0/minicluster-data/master-1/wal/instance:
uuid: "81a36fa646864b2080b554566106f120"
format_stamp: "Formatted at 2026-08-08 22:26:23 on dist-test-slave-ncp9"
server_key: "1eecd907cb2ca7fca91e95c9403c1ed0"
server_key_iv: "ddfa298a8acfd834983c3331db05346f"
server_key_version: "encryptionkey@0"
I20260808 22:26:23.534111 11064 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 22:26:23.536394 11094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:26:23.537689 11064 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 22:26:23.537806 11064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786227706282457-9923-0/minicluster-data/master-1/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786227706282457-9923-0/minicluster-data/master-1/wal
uuid: "81a36fa646864b2080b554566106f120"
format_stamp: "Formatted at 2026-08-08 22:26:23 on dist-test-slave-ncp9"
server_key: "1eecd907cb2ca7fca91e95c9403c1ed0"
server_key_iv: "ddfa298a8acfd834983c3331db05346f"
server_key_version: "encryptionkey@0"
I20260808 22:26:23.537896 11064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786227706282457-9923-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786227706282457-9923-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786227706282457-9923-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:26:23.548359 11064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:26:23.551265 11064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:26:23.551429 11064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:26:23.558264 11064 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.253:42637
I20260808 22:26:23.558535 11172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.253:42637 every 8 connection(s)
I20260808 22:26:23.559172 11064 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786227706282457-9923-0/minicluster-data/master-1/data/info.pb
I20260808 22:26:23.561825 11173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:26:23.563619 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 11064
I20260808 22:26:23.563740 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786227706282457-9923-0/minicluster-data/master-1/wal/instance
I20260808 22:26:23.564013 9923 external_mini_cluster.cc:1428] Setting key 34c6f32de1068dd68334bfe36a1634fa
I20260808 22:26:23.565632 11173 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.176.254" port: 35165 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.9.176.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.176.252@KRBTEST.COM" created.
Aug 08 22:26:23 dist-test-slave-ncp9 krb5kdc[10921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227983, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.253@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
I20260808 22:26:23.579046 11055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:23.567402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53098 (local address 127.9.176.254:35165)
0808 22:26:23.567548 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0808 22:26:23.567553 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:26:23.568391 (+ 838us) server_negotiation.cc:407] Connection header received
0808 22:26:23.569418 (+ 1027us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:26:23.569422 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:26:23.569483 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:26:23.569567 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:26:23.570702 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:23.571237 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:23.572391 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:23.572640 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:23.572730 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:26:23.575439 (+ 2709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:26:23.575443 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:26:23.575445 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:26:23.575474 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:26:23.576936 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:26:23.577506 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:26:23.577509 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:26:23.577510 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:26:23.577581 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:26:23.577912 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:26:23.577914 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:26:23.577915 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:26:23.578111 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:26:23.578179 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0808 22:26:23.578784 (+ 605us) server_negotiation.cc:299] Negotiation successful
0808 22:26:23.578904 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 22:26:23.580195 11176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:23.567772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:35165 (local address 127.0.0.1:53098)
0808 22:26:23.568241 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0808 22:26:23.568293 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 22:26:23.569214 (+ 921us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:26:23.569739 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:26:23.569745 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:26:23.570092 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:26:23.570561 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:26:23.570574 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:26:23.571356 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:26:23.571358 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:26:23.572275 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:26:23.572285 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:26:23.572816 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:26:23.572822 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:26:23.573028 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:26:23.573553 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:26:23.573569 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:26:23.575277 (+ 1708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:26:23.577046 (+ 1769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:26:23.577052 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:26:23.577067 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:26:23.577416 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:26:23.577657 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:26:23.577660 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:26:23.577662 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:26:23.577835 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:26:23.578193 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:26:23.578200 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:26:23.578516 (+ 316us) client_negotiation.cc:769] Sending connection context
0808 22:26:23.578755 (+ 239us) client_negotiation.cc:240] Negotiation successful
0808 22:26:23.579103 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":112,"threads_started":1}
I20260808 22:26:23.581655 11173 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.176.253" port: 42637 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 22:26:23 dist-test-slave-ncp9 krb5kdc[10921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227983, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.253@KRBTEST.COM for kudu/127.9.176.253@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.176.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.252.keytab.
Entry for principal kudu/127.9.176.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.252.keytab.
I20260808 22:26:23.589571 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786227706282457-9923-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786227706282457-9923-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786227706282457-9923-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786227706282457-9923-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.252:46809
--webserver_interface=127.9.176.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.252.keytab
--principal=kudu/127.9.176.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:35643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.176.254:35165,127.9.176.253:42637,127.9.176.252:46809
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 22:26:23.590046 11176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:23.582146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.253:42637 (local address 127.0.0.1:59624)
0808 22:26:23.582270 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0808 22:26:23.582283 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 22:26:23.582347 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:26:23.582886 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:26:23.582890 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:26:23.583060 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:26:23.583177 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:26:23.583184 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:26:23.584144 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:26:23.584147 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:26:23.584693 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:26:23.584702 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:26:23.585351 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:26:23.585355 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:26:23.585449 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:26:23.585817 (+ 368us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:26:23.585827 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:26:23.586927 (+ 1100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:26:23.588635 (+ 1708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:26:23.588637 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:26:23.588639 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:26:23.588815 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:26:23.589104 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:26:23.589107 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:26:23.589109 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:26:23.589179 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:26:23.589586 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:26:23.589589 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:26:23.589672 (+ 83us) client_negotiation.cc:769] Sending connection context
0808 22:26:23.589768 (+ 96us) client_negotiation.cc:240] Negotiation successful
0808 22:26:23.589886 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260808 22:26:23.590229 11179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:23.582276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59624 (local address 127.9.176.253:42637)
0808 22:26:23.582568 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0808 22:26:23.582576 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 22:26:23.582592 (+ 16us) server_negotiation.cc:407] Connection header received
0808 22:26:23.582621 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:26:23.582625 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:26:23.582759 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:26:23.582864 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:26:23.583279 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:23.584031 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:23.584809 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:23.585226 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:23.585308 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:26:23.587062 (+ 1754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:26:23.587071 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:26:23.587080 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:26:23.587101 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:26:23.588531 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:26:23.588940 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:26:23.588945 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:26:23.588946 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:26:23.589021 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:26:23.589257 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:26:23.589260 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:26:23.589261 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:26:23.589486 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:26:23.589604 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0808 22:26:23.589958 (+ 354us) server_negotiation.cc:299] Negotiation successful
0808 22:26:23.590091 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":117,"threads_started":1}
I20260808 22:26:23.592025 11173 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.176.252" port: 46809 } has no permanent_uuid. Determining permanent_uuid...
I20260808 22:26:23.592682 11176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:23.592419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.252:46809 (local address 127.0.0.1:55836)
0808 22:26:23.592537 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0808 22:26:23.592616 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.252:46809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260808 22:26:23.592867 11098 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.176.252:46809: connect: Connection refused (error 111)
I20260808 22:26:23.593710 11176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:23.593495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.252:46809 (local address 127.0.0.1:55840)
0808 22:26:23.593595 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0808 22:26:23.593647 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.252:46809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260808 22:26:23.593856 11173 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.176.252:46809: Network error: Client connection negotiation failed: client connection to 127.9.176.252:46809: connect: Connection refused (error 111)
I20260808 22:26:23.595778 11046 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.176.253" port: 42637 } attempt: 3
Aug 08 22:26:23 dist-test-slave-ncp9 krb5kdc[10921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227983, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.254@KRBTEST.COM for kudu/127.9.176.253@KRBTEST.COM
I20260808 22:26:23.604099 11056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:23.596143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.253:42637 (local address 127.0.0.1:59636)
0808 22:26:23.596321 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0808 22:26:23.596334 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 22:26:23.596409 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:26:23.596607 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:26:23.596609 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:26:23.596742 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:26:23.596838 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:26:23.596843 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:26:23.597803 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:26:23.597805 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:26:23.598738 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:26:23.598747 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:26:23.599307 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:26:23.599309 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:26:23.599384 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:26:23.599808 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:26:23.599819 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:26:23.601142 (+ 1323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:26:23.602845 (+ 1703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:26:23.602847 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:26:23.602849 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:26:23.603047 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:26:23.603299 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:26:23.603302 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:26:23.603304 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:26:23.603390 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:26:23.603730 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:26:23.603732 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:26:23.603808 (+ 76us) client_negotiation.cc:769] Sending connection context
0808 22:26:23.603879 (+ 71us) client_negotiation.cc:240] Negotiation successful
0808 22:26:23.603985 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109}
I20260808 22:26:23.604192 11179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:23.596229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59636 (local address 127.9.176.253:42637)
0808 22:26:23.596338 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0808 22:26:23.596342 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:26:23.596375 (+ 33us) server_negotiation.cc:407] Connection header received
0808 22:26:23.596469 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:26:23.596472 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:26:23.596515 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:26:23.596598 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:26:23.596959 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:23.597706 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:23.598851 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:23.599208 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:23.599317 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:26:23.601274 (+ 1957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:26:23.601278 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:26:23.601280 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:26:23.601305 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:26:23.602733 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:26:23.603124 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:26:23.603127 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:26:23.603129 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:26:23.603207 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:26:23.603460 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:26:23.603463 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:26:23.603465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:26:23.603655 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:26:23.603745 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0808 22:26:23.603937 (+ 192us) server_negotiation.cc:299] Negotiation successful
0808 22:26:23.604052 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260808 22:26:23.605163 11046 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.176.252" port: 46809 } has no permanent_uuid. Determining permanent_uuid...
I20260808 22:26:23.605760 11056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:23.605558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.252:46809 (local address 127.0.0.1:55846)
0808 22:26:23.605656 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0808 22:26:23.605704 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.252:46809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260808 22:26:23.606427 11056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:23.606203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.252:46809 (local address 127.0.0.1:55862)
0808 22:26:23.606342 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0808 22:26:23.606382 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.252:46809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260808 22:26:23.606534 11046 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.176.252:46809: Network error: Client connection negotiation failed: client connection to 127.9.176.252:46809: connect: Connection refused (error 111)
I20260808 22:26:23.637674 11046 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.176.252" port: 46809 } attempt: 1
I20260808 22:26:23.638557 11056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:23.638257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.252:46809 (local address 127.0.0.1:55872)
0808 22:26:23.638408 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0808 22:26:23.638476 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.252:46809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260808 22:26:23.639223 11056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:23.639036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.252:46809 (local address 127.0.0.1:55874)
0808 22:26:23.639119 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0808 22:26:23.639169 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.252:46809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260808 22:26:23.639343 11046 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.176.252:46809: Network error: Client connection negotiation failed: client connection to 127.9.176.252:46809: connect: Connection refused (error 111)
I20260808 22:26:23.643072 11173 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.176.252" port: 46809 } attempt: 1
I20260808 22:26:23.643744 11176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:23.643494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.252:46809 (local address 127.0.0.1:55888)
0808 22:26:23.643618 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0808 22:26:23.643671 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.252:46809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260808 22:26:23.644423 11176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:23.644214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.252:46809 (local address 127.0.0.1:55898)
0808 22:26:23.644324 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0808 22:26:23.644369 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.252:46809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260808 22:26:23.644562 11173 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.176.252:46809: Network error: Client connection negotiation failed: client connection to 127.9.176.252:46809: connect: Connection refused (error 111)
W20260808 22:26:23.703586 11180 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 22:26:23.703877 11180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:26:23.703912 11180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:26:23.703969 11180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:26:23.707904 11180 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:26:23.707962 11180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:26:23.707988 11180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:26:23.708009 11180 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:26:23.708029 11180 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:26:23.712718 11173 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.176.252" port: 46809 } attempt: 2
I20260808 22:26:23.713447 11180 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:35643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786227706282457-9923-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786227706282457-9923-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.9.176.254:35165,127.9.176.253:42637,127.9.176.252:46809
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.252.keytab
--principal=kudu/127.9.176.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.252:46809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786227706282457-9923-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.9.176.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11180
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786227706282457-9923-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:26:23.713618 11176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:23.713301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.252:46809 (local address 127.0.0.1:55906)
0808 22:26:23.713461 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0808 22:26:23.713532 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.252:46809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260808 22:26:23.714362 11176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:23.714125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.252:46809 (local address 127.0.0.1:55916)
0808 22:26:23.714249 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0808 22:26:23.714300 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.252:46809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260808 22:26:23.714517 11173 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.176.252:46809: Network error: Client connection negotiation failed: client connection to 127.9.176.252:46809: connect: Connection refused (error 111)
I20260808 22:26:23.714497 11046 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.176.252" port: 46809 } attempt: 2
I20260808 22:26:23.714726 11180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:26:23.715137 11056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:23.714888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.252:46809 (local address 127.0.0.1:55918)
0808 22:26:23.715016 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0808 22:26:23.715071 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.252:46809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260808 22:26:23.715731 11056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:23.715541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.252:46809 (local address 127.0.0.1:55932)
0808 22:26:23.715639 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0808 22:26:23.715682 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.252:46809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260808 22:26:23.715873 11046 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.176.252:46809: Network error: Client connection negotiation failed: client connection to 127.9.176.252:46809: connect: Connection refused (error 111)
I20260808 22:26:23.715878 11180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:26:23.724084 11189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:26:23.724099 11190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:26:23.724079 11192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:26:23.724402 11180 server_base.cc:1034] running on GCE node
I20260808 22:26:23.725072 11180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:26:23.726141 11180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:26:23.727309 11180 hybrid_clock.cc:648] HybridClock initialized: now 1786227983727288 us; error 29 us; skew 500 ppm
Aug 08 22:26:23 dist-test-slave-ncp9 krb5kdc[10921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227983, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:26:23.730008 11180 init.cc:377] Logged in from keytab as kudu/127.9.176.252@KRBTEST.COM (short username kudu)
I20260808 22:26:23.730875 11180 webserver.cc:460] Webserver started at http://127.9.176.252:36263/ using document root <none> and password file <none>
I20260808 22:26:23.731360 11180 fs_manager.cc:359] Metadata directory not provided
I20260808 22:26:23.731415 11180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:26:23.731585 11180 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:26:23.733582 11180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786227706282457-9923-0/minicluster-data/master-2/data/instance:
uuid: "fcb7dd22db7142e8a944cf470c55e43e"
format_stamp: "Formatted at 2026-08-08 22:26:23 on dist-test-slave-ncp9"
server_key: "0d5152ba9b7bc6249f47ab2dd07ea0a8"
server_key_iv: "8b66ba6dac156f6cecb3afb37d147a8d"
server_key_version: "encryptionkey@0"
I20260808 22:26:23.734078 11180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786227706282457-9923-0/minicluster-data/master-2/wal/instance:
uuid: "fcb7dd22db7142e8a944cf470c55e43e"
format_stamp: "Formatted at 2026-08-08 22:26:23 on dist-test-slave-ncp9"
server_key: "0d5152ba9b7bc6249f47ab2dd07ea0a8"
server_key_iv: "8b66ba6dac156f6cecb3afb37d147a8d"
server_key_version: "encryptionkey@0"
I20260808 22:26:23.737654 11180 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 22:26:23.740074 11200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:26:23.741335 11180 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 22:26:23.741499 11180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786227706282457-9923-0/minicluster-data/master-2/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786227706282457-9923-0/minicluster-data/master-2/wal
uuid: "fcb7dd22db7142e8a944cf470c55e43e"
format_stamp: "Formatted at 2026-08-08 22:26:23 on dist-test-slave-ncp9"
server_key: "0d5152ba9b7bc6249f47ab2dd07ea0a8"
server_key_iv: "8b66ba6dac156f6cecb3afb37d147a8d"
server_key_version: "encryptionkey@0"
I20260808 22:26:23.741608 11180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786227706282457-9923-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786227706282457-9923-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786227706282457-9923-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:26:23.757755 11180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:26:23.760391 11180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:26:23.760586 11180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:26:23.767881 11180 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.252:46809
I20260808 22:26:23.768131 11283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.252:46809 every 8 connection(s)
I20260808 22:26:23.768841 11180 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786227706282457-9923-0/minicluster-data/master-2/data/info.pb
I20260808 22:26:23.771483 11284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:26:23.775013 11284 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.176.254" port: 35165 } has no permanent_uuid. Determining permanent_uuid...
I20260808 22:26:23.775904 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 11180
I20260808 22:26:23.776036 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786227706282457-9923-0/minicluster-data/master-2/wal/instance
I20260808 22:26:23.776320 9923 external_mini_cluster.cc:1428] Setting key 277b7890b151ec0eb56d8107fa548a82
Aug 08 22:26:23 dist-test-slave-ncp9 krb5kdc[10921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227983, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.252@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
Aug 08 22:26:23 dist-test-slave-ncp9 krb5kdc[10921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.176.252@KRBTEST.COM
I20260808 22:26:23.787672 11055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:23.776961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53110 (local address 127.9.176.254:35165)
0808 22:26:23.777088 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0808 22:26:23.777093 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:26:23.778047 (+ 954us) server_negotiation.cc:407] Connection header received
0808 22:26:23.778716 (+ 669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:26:23.778719 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:26:23.778767 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:26:23.778851 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:26:23.780098 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:23.780580 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:23.781550 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:23.781757 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:23.781844 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:26:23.784150 (+ 2306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:26:23.784153 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:26:23.784155 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:26:23.784179 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:26:23.785612 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:26:23.786137 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:26:23.786140 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:26:23.786141 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:26:23.786228 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:26:23.786545 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:26:23.786549 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:26:23.786551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:26:23.786767 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:26:23.786865 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0808 22:26:23.787389 (+ 524us) server_negotiation.cc:299] Negotiation successful
0808 22:26:23.787513 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 22:26:23.788578 11287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:23.777395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55944 (local address 127.9.176.252:46809)
0808 22:26:23.777786 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0808 22:26:23.777804 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0808 22:26:23.777835 (+ 31us) server_negotiation.cc:407] Connection header received
0808 22:26:23.778503 (+ 668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:26:23.778529 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:26:23.778727 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:26:23.778895 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:26:23.779644 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:23.780642 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:23.781516 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:23.781880 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:23.782022 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:26:23.784548 (+ 2526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:26:23.784555 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:26:23.784566 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:26:23.784592 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:26:23.786203 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:26:23.786647 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:26:23.786651 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:26:23.786653 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:26:23.786721 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:26:23.786987 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:26:23.786990 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:26:23.786991 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:26:23.787247 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:26:23.787412 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0808 22:26:23.787672 (+ 260us) server_negotiation.cc:299] Negotiation successful
0808 22:26:23.787791 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":265,"spinlock_wait_cycles":4992,"thread_start_us":115,"threads_started":1}
I20260808 22:26:23.788578 11288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:23.777395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:35165 (local address 127.0.0.1:53110)
0808 22:26:23.777877 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0808 22:26:23.777925 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0808 22:26:23.778508 (+ 583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:26:23.779482 (+ 974us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:26:23.779486 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:26:23.779806 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:26:23.779956 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:26:23.779966 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:26:23.780690 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:26:23.780692 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:26:23.781435 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:26:23.781445 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:26:23.781885 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:26:23.781889 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:26:23.782018 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:26:23.782543 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:26:23.782558 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:26:23.784019 (+ 1461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:26:23.785717 (+ 1698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:26:23.785725 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:26:23.785734 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:26:23.786019 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:26:23.786327 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:26:23.786330 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:26:23.786332 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:26:23.786457 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:26:23.786866 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:26:23.786871 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:26:23.787133 (+ 262us) client_negotiation.cc:769] Sending connection context
0808 22:26:23.787328 (+ 195us) client_negotiation.cc:240] Negotiation successful
0808 22:26:23.787624 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"mutex_wait_us":1,"spinlock_wait_cycles":42368,"thread_start_us":141,"threads_started":1}
I20260808 22:26:23.789781 11284 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.176.253" port: 42637 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 22:26:23 dist-test-slave-ncp9 krb5kdc[10921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227983, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.252@KRBTEST.COM for kudu/127.9.176.253@KRBTEST.COM
Aug 08 22:26:23 dist-test-slave-ncp9 krb5kdc[10921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
I20260808 22:26:23.797644 11288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:23.790225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.253:42637 (local address 127.0.0.1:59640)
0808 22:26:23.790374 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0808 22:26:23.790386 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 22:26:23.790447 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:26:23.790659 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:26:23.790661 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:26:23.790807 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:26:23.790916 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:26:23.790923 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:26:23.791677 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:26:23.791680 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:26:23.792197 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:26:23.792205 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:26:23.792628 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:26:23.792632 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:26:23.792723 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:26:23.793128 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:26:23.793141 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:26:23.794495 (+ 1354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:26:23.796164 (+ 1669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:26:23.796167 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:26:23.796169 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:26:23.796385 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:26:23.796719 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:26:23.796722 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:26:23.796724 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:26:23.796816 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:26:23.797192 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:26:23.797195 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:26:23.797300 (+ 105us) client_negotiation.cc:769] Sending connection context
0808 22:26:23.797407 (+ 107us) client_negotiation.cc:240] Negotiation successful
0808 22:26:23.797536 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90}
I20260808 22:26:23.797734 11179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:23.790353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59640 (local address 127.9.176.253:42637)
0808 22:26:23.790464 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0808 22:26:23.790468 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:26:23.790482 (+ 14us) server_negotiation.cc:407] Connection header received
0808 22:26:23.790523 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:26:23.790526 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:26:23.790572 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:26:23.790654 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:26:23.791084 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:23.791576 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:23.792304 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:23.792533 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:23.792615 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:26:23.794626 (+ 2011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:26:23.794629 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:26:23.794631 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:26:23.794655 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:26:23.796063 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:26:23.796513 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:26:23.796516 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:26:23.796517 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:26:23.796620 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:26:23.796920 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:26:23.796923 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:26:23.796925 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:26:23.797099 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:26:23.797172 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0808 22:26:23.797442 (+ 270us) server_negotiation.cc:299] Negotiation successful
0808 22:26:23.797579 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260808 22:26:23.798421 11284 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.176.252" port: 46809 } has no permanent_uuid. Determining permanent_uuid...
I20260808 22:26:23.799116 11055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:23.790965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53114 (local address 127.9.176.254:35165)
0808 22:26:23.791065 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0808 22:26:23.791068 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:26:23.791080 (+ 12us) server_negotiation.cc:407] Connection header received
0808 22:26:23.791171 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:26:23.791174 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:26:23.791210 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:26:23.791273 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:26:23.791833 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:23.792296 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:23.793064 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:23.793278 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:23.793358 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:26:23.795950 (+ 2592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:26:23.795952 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:26:23.795954 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:26:23.795976 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:26:23.797399 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:26:23.797937 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:26:23.797940 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:26:23.797941 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:26:23.798024 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:26:23.798277 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:26:23.798280 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:26:23.798281 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:26:23.798472 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:26:23.798714 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0808 22:26:23.798790 (+ 76us) server_negotiation.cc:299] Negotiation successful
0808 22:26:23.798887 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
Aug 08 22:26:23 dist-test-slave-ncp9 krb5kdc[10921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227983, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.252@KRBTEST.COM for kudu/127.9.176.252@KRBTEST.COM
Aug 08 22:26:23 dist-test-slave-ncp9 krb5kdc[10921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.176.253@KRBTEST.COM
I20260808 22:26:23.805686 11173 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.176.252" port: 46809 } attempt: 3
I20260808 22:26:23.805774 11288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:23.798843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.252:46809 (local address 127.0.0.1:55948)
0808 22:26:23.798924 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0808 22:26:23.798936 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 22:26:23.799014 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:26:23.799197 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:26:23.799199 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:26:23.799333 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:26:23.799392 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:26:23.799396 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:26:23.800242 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:26:23.800244 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:26:23.800739 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:26:23.800747 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:26:23.801292 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:26:23.801295 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:26:23.801375 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:26:23.801747 (+ 372us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:26:23.801759 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:26:23.802860 (+ 1101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:26:23.804417 (+ 1557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:26:23.804420 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:26:23.804421 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:26:23.804593 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:26:23.804855 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:26:23.804858 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:26:23.804859 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:26:23.804972 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:26:23.805293 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:26:23.805296 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:26:23.805377 (+ 81us) client_negotiation.cc:769] Sending connection context
0808 22:26:23.805478 (+ 101us) client_negotiation.cc:240] Negotiation successful
0808 22:26:23.805597 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260808 22:26:23.805925 11287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:23.798957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55948 (local address 127.9.176.252:46809)
0808 22:26:23.799040 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0808 22:26:23.799043 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:26:23.799054 (+ 11us) server_negotiation.cc:407] Connection header received
0808 22:26:23.799082 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:26:23.799084 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:26:23.799116 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:26:23.799176 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:26:23.799498 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:23.800135 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:23.800844 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:23.801192 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:23.801272 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:26:23.802974 (+ 1702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:26:23.802978 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:26:23.802980 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:26:23.803002 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:26:23.804325 (+ 1323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:26:23.804679 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:26:23.804682 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:26:23.804684 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:26:23.804767 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:26:23.805050 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:26:23.805052 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:26:23.805054 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:26:23.805209 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:26:23.805289 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0808 22:26:23.805550 (+ 261us) server_negotiation.cc:299] Negotiation successful
0808 22:26:23.805659 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260808 22:26:23.808991 11179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:23.800234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59644 (local address 127.9.176.253:42637)
0808 22:26:23.800331 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0808 22:26:23.800334 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:26:23.800346 (+ 12us) server_negotiation.cc:407] Connection header received
0808 22:26:23.800436 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:26:23.800439 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:26:23.800476 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:26:23.800554 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:26:23.801182 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:23.801833 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:23.802554 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:23.802814 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:23.802900 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:26:23.805172 (+ 2272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:26:23.805176 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:26:23.805178 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:26:23.805202 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:26:23.807075 (+ 1873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:26:23.807420 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:26:23.807423 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:26:23.807425 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:26:23.807498 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:26:23.807739 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:26:23.807742 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:26:23.807744 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:26:23.807956 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:26:23.808171 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0808 22:26:23.808709 (+ 538us) server_negotiation.cc:299] Negotiation successful
0808 22:26:23.808807 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":40}
I20260808 22:26:23.809415 11284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcb7dd22db7142e8a944cf470c55e43e: Bootstrap starting.
Aug 08 22:26:23 dist-test-slave-ncp9 krb5kdc[10921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227983, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.253@KRBTEST.COM for kudu/127.9.176.252@KRBTEST.COM
I20260808 22:26:23.812258 11284 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fcb7dd22db7142e8a944cf470c55e43e: Neither blocks nor log segments found. Creating new log.
I20260808 22:26:23.812651 9923 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 22:26:23.813174 11284 log.cc:824] T 00000000000000000000000000000000 P fcb7dd22db7142e8a944cf470c55e43e: Log is configured to *not* fsync() on all Append() calls
I20260808 22:26:23.814230 11287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:23.806332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55964 (local address 127.9.176.252:46809)
0808 22:26:23.806419 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0808 22:26:23.806423 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:26:23.806435 (+ 12us) server_negotiation.cc:407] Connection header received
0808 22:26:23.806464 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:26:23.806466 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:26:23.806497 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:26:23.806575 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:26:23.807046 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:23.807654 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:23.808672 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:23.809012 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:23.809417 (+ 405us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:26:23.811056 (+ 1639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:26:23.811060 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:26:23.811062 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:26:23.811084 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:26:23.812579 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:26:23.813091 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:26:23.813095 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:26:23.813096 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:26:23.813177 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:26:23.813448 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:26:23.813452 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:26:23.813453 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:26:23.813674 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:26:23.813764 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0808 22:26:23.814047 (+ 283us) server_negotiation.cc:299] Negotiation successful
0808 22:26:23.814135 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260808 22:26:23.814245 11176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:23.806123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.252:46809 (local address 127.0.0.1:55964)
0808 22:26:23.806225 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0808 22:26:23.806237 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 22:26:23.806304 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:26:23.806596 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:26:23.806599 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:26:23.806830 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:26:23.806927 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:26:23.806934 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:26:23.807790 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:26:23.807794 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:26:23.808548 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:26:23.808559 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:26:23.809148 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:26:23.809152 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:26:23.809229 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:26:23.809650 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:26:23.809664 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:26:23.810930 (+ 1266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:26:23.812710 (+ 1780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:26:23.812713 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:26:23.812715 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:26:23.812981 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:26:23.813268 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:26:23.813271 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:26:23.813273 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:26:23.813358 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:26:23.813774 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:26:23.813777 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:26:23.813871 (+ 94us) client_negotiation.cc:769] Sending connection context
0808 22:26:23.813973 (+ 102us) client_negotiation.cc:240] Negotiation successful
0808 22:26:23.814106 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":18}
I20260808 22:26:23.815011 11046 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.176.252" port: 46809 } attempt: 3
I20260808 22:26:23.814101 11004 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "f180cbc2339f4ee8abe6d13f52d13268"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53114
I20260808 22:26:23.816184 11284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcb7dd22db7142e8a944cf470c55e43e: No bootstrap required, opened a new log
I20260808 22:26:23.817194 11173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81a36fa646864b2080b554566106f120: Bootstrap starting.
I20260808 22:26:23.819454 11173 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 81a36fa646864b2080b554566106f120: Neither blocks nor log segments found. Creating new log.
I20260808 22:26:23.820091 11173 log.cc:824] T 00000000000000000000000000000000 P 81a36fa646864b2080b554566106f120: Log is configured to *not* fsync() on all Append() calls
Aug 08 22:26:23 dist-test-slave-ncp9 krb5kdc[10921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227983, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.254@KRBTEST.COM for kudu/127.9.176.252@KRBTEST.COM
I20260808 22:26:23.820284 11284 raft_consensus.cc:348] T 00000000000000000000000000000000 P fcb7dd22db7142e8a944cf470c55e43e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f180cbc2339f4ee8abe6d13f52d13268" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 35165 } } peers { permanent_uuid: "81a36fa646864b2080b554566106f120" member_type: VOTER last_known_addr { host: "127.9.176.253" port: 42637 } } peers { permanent_uuid: "fcb7dd22db7142e8a944cf470c55e43e" member_type: VOTER last_known_addr { host: "127.9.176.252" port: 46809 } }
I20260808 22:26:23.820541 11284 raft_consensus.cc:729] T 00000000000000000000000000000000 P fcb7dd22db7142e8a944cf470c55e43e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcb7dd22db7142e8a944cf470c55e43e, State: Initialized, Role: FOLLOWER
I20260808 22:26:23.820884 11284 consensus_queue.cc:260] T 00000000000000000000000000000000 P fcb7dd22db7142e8a944cf470c55e43e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f180cbc2339f4ee8abe6d13f52d13268" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 35165 } } peers { permanent_uuid: "81a36fa646864b2080b554566106f120" member_type: VOTER last_known_addr { host: "127.9.176.253" port: 42637 } } peers { permanent_uuid: "fcb7dd22db7142e8a944cf470c55e43e" member_type: VOTER last_known_addr { host: "127.9.176.252" port: 46809 } }
I20260808 22:26:23.821409 11291 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcb7dd22db7142e8a944cf470c55e43e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f180cbc2339f4ee8abe6d13f52d13268" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 35165 } } peers { permanent_uuid: "81a36fa646864b2080b554566106f120" member_type: VOTER last_known_addr { host: "127.9.176.253" port: 42637 } } peers { permanent_uuid: "fcb7dd22db7142e8a944cf470c55e43e" member_type: VOTER last_known_addr { host: "127.9.176.252" port: 46809 } } }
I20260808 22:26:23.821794 11291 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcb7dd22db7142e8a944cf470c55e43e [sys.catalog]: This master's current role is: FOLLOWER
I20260808 22:26:23.822289 11173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81a36fa646864b2080b554566106f120: No bootstrap required, opened a new log
I20260808 22:26:23.822355 11284 sys_catalog.cc:564] T 00000000000000000000000000000000 P fcb7dd22db7142e8a944cf470c55e43e [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:26:23.824209 11056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:23.815350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.252:46809 (local address 127.0.0.1:55974)
0808 22:26:23.815517 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0808 22:26:23.815530 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 22:26:23.815598 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:26:23.816097 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:26:23.816100 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:26:23.816268 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:26:23.816367 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:26:23.816374 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:26:23.817294 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:26:23.817298 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:26:23.818084 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:26:23.818095 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:26:23.818625 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:26:23.818629 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:26:23.818714 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:26:23.819169 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:26:23.819184 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:26:23.820564 (+ 1380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:26:23.822571 (+ 2007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:26:23.822574 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:26:23.822576 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:26:23.822834 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:26:23.823149 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:26:23.823151 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:26:23.823153 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:26:23.823248 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:26:23.823701 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:26:23.823704 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:26:23.823803 (+ 99us) client_negotiation.cc:769] Sending connection context
0808 22:26:23.823898 (+ 95us) client_negotiation.cc:240] Negotiation successful
0808 22:26:23.824048 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94}
I20260808 22:26:23.824740 11287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:23.815783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55974 (local address 127.9.176.252:46809)
0808 22:26:23.815886 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0808 22:26:23.815890 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:26:23.815902 (+ 12us) server_negotiation.cc:407] Connection header received
0808 22:26:23.815934 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:26:23.815936 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:26:23.815972 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:26:23.816041 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:26:23.816498 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:23.817189 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:23.818212 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:23.818525 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:23.818622 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:26:23.820744 (+ 2122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:26:23.820747 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:26:23.820750 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:26:23.820773 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:26:23.822438 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:26:23.822957 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:26:23.822961 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:26:23.822963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:26:23.823045 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:26:23.823353 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:26:23.823356 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:26:23.823358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:26:23.823561 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:26:23.823650 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0808 22:26:23.824502 (+ 852us) server_negotiation.cc:299] Negotiation successful
0808 22:26:23.824612 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260808 22:26:23.826305 11173 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81a36fa646864b2080b554566106f120 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f180cbc2339f4ee8abe6d13f52d13268" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 35165 } } peers { permanent_uuid: "81a36fa646864b2080b554566106f120" member_type: VOTER last_known_addr { host: "127.9.176.253" port: 42637 } } peers { permanent_uuid: "fcb7dd22db7142e8a944cf470c55e43e" member_type: VOTER last_known_addr { host: "127.9.176.252" port: 46809 } }
I20260808 22:26:23.826510 11173 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81a36fa646864b2080b554566106f120 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81a36fa646864b2080b554566106f120, State: Initialized, Role: FOLLOWER
I20260808 22:26:23.826829 11173 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81a36fa646864b2080b554566106f120 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f180cbc2339f4ee8abe6d13f52d13268" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 35165 } } peers { permanent_uuid: "81a36fa646864b2080b554566106f120" member_type: VOTER last_known_addr { host: "127.9.176.253" port: 42637 } } peers { permanent_uuid: "fcb7dd22db7142e8a944cf470c55e43e" member_type: VOTER last_known_addr { host: "127.9.176.252" port: 46809 } }
I20260808 22:26:23.827593 11046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f180cbc2339f4ee8abe6d13f52d13268: Bootstrap starting.
I20260808 22:26:23.828135 11173 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81a36fa646864b2080b554566106f120 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:26:23.826375 11180 master_runner.cc:186] Error getting master registration for 127.9.176.254:35165: OK, instance_id { permanent_uuid: "f180cbc2339f4ee8abe6d13f52d13268" instance_seqno: 1786227983355715 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 22:26:23.830211 11180 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 22:26:23.830236 11046 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f180cbc2339f4ee8abe6d13f52d13268: Neither blocks nor log segments found. Creating new log.
W20260808 22:26:23.830363 11311 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fcb7dd22db7142e8a944cf470c55e43e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260808 22:26:23.830461 11312 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81a36fa646864b2080b554566106f120 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f180cbc2339f4ee8abe6d13f52d13268" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 35165 } } peers { permanent_uuid: "81a36fa646864b2080b554566106f120" member_type: VOTER last_known_addr { host: "127.9.176.253" port: 42637 } } peers { permanent_uuid: "fcb7dd22db7142e8a944cf470c55e43e" member_type: VOTER last_known_addr { host: "127.9.176.252" port: 46809 } } }
I20260808 22:26:23.830595 11312 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81a36fa646864b2080b554566106f120 [sys.catalog]: This master's current role is: FOLLOWER
W20260808 22:26:23.830688 11311 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 22:26:23.831117 11046 log.cc:824] T 00000000000000000000000000000000 P f180cbc2339f4ee8abe6d13f52d13268: Log is configured to *not* fsync() on all Append() calls
W20260808 22:26:23.833735 11325 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 81a36fa646864b2080b554566106f120: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 22:26:23.833802 11325 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 22:26:23.833847 11046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f180cbc2339f4ee8abe6d13f52d13268: No bootstrap required, opened a new log
I20260808 22:26:23.831475 11064 master_runner.cc:186] Error getting master registration for 127.9.176.254:35165: OK, instance_id { permanent_uuid: "f180cbc2339f4ee8abe6d13f52d13268" instance_seqno: 1786227983355715 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 22:26:23.835166 11064 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 22:26:23.838179 11046 raft_consensus.cc:348] T 00000000000000000000000000000000 P f180cbc2339f4ee8abe6d13f52d13268 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f180cbc2339f4ee8abe6d13f52d13268" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 35165 } } peers { permanent_uuid: "81a36fa646864b2080b554566106f120" member_type: VOTER last_known_addr { host: "127.9.176.253" port: 42637 } } peers { permanent_uuid: "fcb7dd22db7142e8a944cf470c55e43e" member_type: VOTER last_known_addr { host: "127.9.176.252" port: 46809 } }
I20260808 22:26:23.838505 11046 raft_consensus.cc:729] T 00000000000000000000000000000000 P f180cbc2339f4ee8abe6d13f52d13268 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f180cbc2339f4ee8abe6d13f52d13268, State: Initialized, Role: FOLLOWER
I20260808 22:26:23.838963 11046 consensus_queue.cc:260] T 00000000000000000000000000000000 P f180cbc2339f4ee8abe6d13f52d13268 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f180cbc2339f4ee8abe6d13f52d13268" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 35165 } } peers { permanent_uuid: "81a36fa646864b2080b554566106f120" member_type: VOTER last_known_addr { host: "127.9.176.253" port: 42637 } } peers { permanent_uuid: "fcb7dd22db7142e8a944cf470c55e43e" member_type: VOTER last_known_addr { host: "127.9.176.252" port: 46809 } }
I20260808 22:26:23.839519 11329 sys_catalog.cc:455] T 00000000000000000000000000000000 P f180cbc2339f4ee8abe6d13f52d13268 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f180cbc2339f4ee8abe6d13f52d13268" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 35165 } } peers { permanent_uuid: "81a36fa646864b2080b554566106f120" member_type: VOTER last_known_addr { host: "127.9.176.253" port: 42637 } } peers { permanent_uuid: "fcb7dd22db7142e8a944cf470c55e43e" member_type: VOTER last_known_addr { host: "127.9.176.252" port: 46809 } } }
I20260808 22:26:23.839704 11329 sys_catalog.cc:458] T 00000000000000000000000000000000 P f180cbc2339f4ee8abe6d13f52d13268 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 22:26:23.840931 11004 raft_consensus.cc:482] T 00000000000000000000000000000000 P f180cbc2339f4ee8abe6d13f52d13268 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 22:26:23.841073 11004 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f180cbc2339f4ee8abe6d13f52d13268 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:26:23.842371 11004 raft_consensus.cc:504] T 00000000000000000000000000000000 P f180cbc2339f4ee8abe6d13f52d13268 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f180cbc2339f4ee8abe6d13f52d13268" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 35165 } } peers { permanent_uuid: "81a36fa646864b2080b554566106f120" member_type: VOTER last_known_addr { host: "127.9.176.253" port: 42637 } } peers { permanent_uuid: "fcb7dd22db7142e8a944cf470c55e43e" member_type: VOTER last_known_addr { host: "127.9.176.252" port: 46809 } }
I20260808 22:26:23.843267 11004 leader_election.cc:288] T 00000000000000000000000000000000 P f180cbc2339f4ee8abe6d13f52d13268 [CANDIDATE]: Term 1 election: Requested vote from peers 81a36fa646864b2080b554566106f120 (127.9.176.253:42637), fcb7dd22db7142e8a944cf470c55e43e (127.9.176.252:46809)
I20260808 22:26:23.840843 11046 sys_catalog.cc:564] T 00000000000000000000000000000000 P f180cbc2339f4ee8abe6d13f52d13268 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:26:23.844283 11242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f180cbc2339f4ee8abe6d13f52d13268" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "fcb7dd22db7142e8a944cf470c55e43e"
I20260808 22:26:23.844530 11242 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fcb7dd22db7142e8a944cf470c55e43e [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:26:23.845321 11134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f180cbc2339f4ee8abe6d13f52d13268" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "81a36fa646864b2080b554566106f120"
I20260808 22:26:23.845541 11134 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81a36fa646864b2080b554566106f120 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:26:23.845907 11242 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fcb7dd22db7142e8a944cf470c55e43e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f180cbc2339f4ee8abe6d13f52d13268 in term 1.
I20260808 22:26:23.846902 10968 leader_election.cc:302] T 00000000000000000000000000000000 P f180cbc2339f4ee8abe6d13f52d13268 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f180cbc2339f4ee8abe6d13f52d13268, fcb7dd22db7142e8a944cf470c55e43e; no voters:
I20260808 22:26:23.847185 11329 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f180cbc2339f4ee8abe6d13f52d13268 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:26:23.847844 11329 raft_consensus.cc:686] T 00000000000000000000000000000000 P f180cbc2339f4ee8abe6d13f52d13268 [term 1 LEADER]: Becoming Leader. State: Replica: f180cbc2339f4ee8abe6d13f52d13268, State: Running, Role: LEADER
I20260808 22:26:23.848189 11329 consensus_queue.cc:237] T 00000000000000000000000000000000 P f180cbc2339f4ee8abe6d13f52d13268 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f180cbc2339f4ee8abe6d13f52d13268" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 35165 } } peers { permanent_uuid: "81a36fa646864b2080b554566106f120" member_type: VOTER last_known_addr { host: "127.9.176.253" port: 42637 } } peers { permanent_uuid: "fcb7dd22db7142e8a944cf470c55e43e" member_type: VOTER last_known_addr { host: "127.9.176.252" port: 46809 } }
I20260808 22:26:23.849952 11134 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 81a36fa646864b2080b554566106f120 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f180cbc2339f4ee8abe6d13f52d13268 in term 1.
W20260808 22:26:23.852428 10973 tablet.cc:2369] T 00000000000000000000000000000000 P f180cbc2339f4ee8abe6d13f52d13268: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 22:26:23.852515 11329 sys_catalog.cc:455] T 00000000000000000000000000000000 P f180cbc2339f4ee8abe6d13f52d13268 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f180cbc2339f4ee8abe6d13f52d13268. Latest consensus state: current_term: 1 leader_uuid: "f180cbc2339f4ee8abe6d13f52d13268" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f180cbc2339f4ee8abe6d13f52d13268" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 35165 } } peers { permanent_uuid: "81a36fa646864b2080b554566106f120" member_type: VOTER last_known_addr { host: "127.9.176.253" port: 42637 } } peers { permanent_uuid: "fcb7dd22db7142e8a944cf470c55e43e" member_type: VOTER last_known_addr { host: "127.9.176.252" port: 46809 } } }
I20260808 22:26:23.852631 11329 sys_catalog.cc:458] T 00000000000000000000000000000000 P f180cbc2339f4ee8abe6d13f52d13268 [sys.catalog]: This master's current role is: LEADER
I20260808 22:26:23.857342 11055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:23.846229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53126 (local address 127.9.176.254:35165)
0808 22:26:23.846493 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0808 22:26:23.846498 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:26:23.847899 (+ 1401us) server_negotiation.cc:407] Connection header received
0808 22:26:23.848104 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:26:23.848108 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:26:23.848156 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:26:23.848316 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:26:23.849715 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:23.850489 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:23.851590 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:23.851894 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:23.851963 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:26:23.853785 (+ 1822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:26:23.853789 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:26:23.853791 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:26:23.853817 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:26:23.855432 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:26:23.855924 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:26:23.855927 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:26:23.855929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:26:23.856015 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:26:23.856302 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:26:23.856305 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:26:23.856307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:26:23.856520 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:26:23.856628 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0808 22:26:23.857050 (+ 422us) server_negotiation.cc:299] Negotiation successful
0808 22:26:23.857165 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260808 22:26:23.857196 11362 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:26:23.860579 11362 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:26:23.864092 11179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:23.847842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59646 (local address 127.9.176.253:42637)
0808 22:26:23.848008 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0808 22:26:23.848012 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:26:23.850335 (+ 2323us) server_negotiation.cc:407] Connection header received
0808 22:26:23.850514 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:26:23.850517 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:26:23.850563 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:26:23.850610 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:26:23.853499 (+ 2889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:23.854172 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:23.857133 (+ 2961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:23.857469 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:23.857551 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:26:23.860233 (+ 2682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:26:23.860237 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:26:23.860240 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:26:23.860265 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:26:23.862029 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:26:23.862828 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:26:23.862831 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:26:23.862833 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:26:23.862916 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:26:23.863189 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:26:23.863193 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:26:23.863195 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:26:23.863391 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:26:23.863506 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0808 22:26:23.863835 (+ 329us) server_negotiation.cc:299] Negotiation successful
0808 22:26:23.863945 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260808 22:26:23.866777 11287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:23.847224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55982 (local address 127.9.176.252:46809)
0808 22:26:23.847333 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0808 22:26:23.847338 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:26:23.848720 (+ 1382us) server_negotiation.cc:407] Connection header received
0808 22:26:23.848920 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:26:23.848923 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:26:23.849039 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:26:23.849114 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:26:23.851526 (+ 2412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:23.852268 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:23.855146 (+ 2878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:23.855444 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:23.855532 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:26:23.858418 (+ 2886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:26:23.858422 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:26:23.858425 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:26:23.858450 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:26:23.860019 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:26:23.860470 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:26:23.862123 (+ 1653us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:26:23.862127 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:26:23.862240 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:26:23.865874 (+ 3634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:26:23.865877 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:26:23.865879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:26:23.866087 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:26:23.866181 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0808 22:26:23.866517 (+ 336us) server_negotiation.cc:299] Negotiation successful
0808 22:26:23.866633 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260808 22:26:23.873785 11134 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 81a36fa646864b2080b554566106f120 [term 1 FOLLOWER]: Refusing update from remote peer f180cbc2339f4ee8abe6d13f52d13268: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:26:23.873795 11242 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fcb7dd22db7142e8a944cf470c55e43e [term 1 FOLLOWER]: Refusing update from remote peer f180cbc2339f4ee8abe6d13f52d13268: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:26:23.874459 11329 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f180cbc2339f4ee8abe6d13f52d13268 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81a36fa646864b2080b554566106f120" member_type: VOTER last_known_addr { host: "127.9.176.253" port: 42637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:26:23.874868 11329 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f180cbc2339f4ee8abe6d13f52d13268 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcb7dd22db7142e8a944cf470c55e43e" member_type: VOTER last_known_addr { host: "127.9.176.252" port: 46809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:26:23.879081 11291 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcb7dd22db7142e8a944cf470c55e43e [sys.catalog]: SysCatalogTable state changed. Reason: New leader f180cbc2339f4ee8abe6d13f52d13268. Latest consensus state: current_term: 1 leader_uuid: "f180cbc2339f4ee8abe6d13f52d13268" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f180cbc2339f4ee8abe6d13f52d13268" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 35165 } } peers { permanent_uuid: "81a36fa646864b2080b554566106f120" member_type: VOTER last_known_addr { host: "127.9.176.253" port: 42637 } } peers { permanent_uuid: "fcb7dd22db7142e8a944cf470c55e43e" member_type: VOTER last_known_addr { host: "127.9.176.252" port: 46809 } } }
I20260808 22:26:23.879081 11312 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81a36fa646864b2080b554566106f120 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f180cbc2339f4ee8abe6d13f52d13268. Latest consensus state: current_term: 1 leader_uuid: "f180cbc2339f4ee8abe6d13f52d13268" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f180cbc2339f4ee8abe6d13f52d13268" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 35165 } } peers { permanent_uuid: "81a36fa646864b2080b554566106f120" member_type: VOTER last_known_addr { host: "127.9.176.253" port: 42637 } } peers { permanent_uuid: "fcb7dd22db7142e8a944cf470c55e43e" member_type: VOTER last_known_addr { host: "127.9.176.252" port: 46809 } } }
I20260808 22:26:23.879184 11291 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcb7dd22db7142e8a944cf470c55e43e [sys.catalog]: This master's current role is: FOLLOWER
I20260808 22:26:23.879184 11312 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81a36fa646864b2080b554566106f120 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 22:26:23.884522 11358 sys_catalog.cc:455] T 00000000000000000000000000000000 P f180cbc2339f4ee8abe6d13f52d13268 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f180cbc2339f4ee8abe6d13f52d13268" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f180cbc2339f4ee8abe6d13f52d13268" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 35165 } } peers { permanent_uuid: "81a36fa646864b2080b554566106f120" member_type: VOTER last_known_addr { host: "127.9.176.253" port: 42637 } } peers { permanent_uuid: "fcb7dd22db7142e8a944cf470c55e43e" member_type: VOTER last_known_addr { host: "127.9.176.252" port: 46809 } } }
I20260808 22:26:23.884631 11358 sys_catalog.cc:458] T 00000000000000000000000000000000 P f180cbc2339f4ee8abe6d13f52d13268 [sys.catalog]: This master's current role is: LEADER
I20260808 22:26:23.885525 11312 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81a36fa646864b2080b554566106f120 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f180cbc2339f4ee8abe6d13f52d13268" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f180cbc2339f4ee8abe6d13f52d13268" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 35165 } } peers { permanent_uuid: "81a36fa646864b2080b554566106f120" member_type: VOTER last_known_addr { host: "127.9.176.253" port: 42637 } } peers { permanent_uuid: "fcb7dd22db7142e8a944cf470c55e43e" member_type: VOTER last_known_addr { host: "127.9.176.252" port: 46809 } } }
I20260808 22:26:23.885625 11312 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81a36fa646864b2080b554566106f120 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 22:26:23.886004 11365 mvcc.cc:205] Tried to move back new op lower bound from 7316389821939752960 to 7316389821846069248. Current Snapshot: MvccSnapshot[applied={T|T < 7316389821939752960}]
I20260808 22:26:23.888278 11370 mvcc.cc:205] Tried to move back new op lower bound from 7316389821939752960 to 7316389821846069248. Current Snapshot: MvccSnapshot[applied={T|T < 7316389821939752960}]
I20260808 22:26:23.888967 11329 sys_catalog.cc:455] T 00000000000000000000000000000000 P f180cbc2339f4ee8abe6d13f52d13268 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f180cbc2339f4ee8abe6d13f52d13268" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f180cbc2339f4ee8abe6d13f52d13268" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 35165 } } peers { permanent_uuid: "81a36fa646864b2080b554566106f120" member_type: VOTER last_known_addr { host: "127.9.176.253" port: 42637 } } peers { permanent_uuid: "fcb7dd22db7142e8a944cf470c55e43e" member_type: VOTER last_known_addr { host: "127.9.176.252" port: 46809 } } }
I20260808 22:26:23.889072 11329 sys_catalog.cc:458] T 00000000000000000000000000000000 P f180cbc2339f4ee8abe6d13f52d13268 [sys.catalog]: This master's current role is: LEADER
I20260808 22:26:23.890060 11291 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcb7dd22db7142e8a944cf470c55e43e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f180cbc2339f4ee8abe6d13f52d13268" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f180cbc2339f4ee8abe6d13f52d13268" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 35165 } } peers { permanent_uuid: "81a36fa646864b2080b554566106f120" member_type: VOTER last_known_addr { host: "127.9.176.253" port: 42637 } } peers { permanent_uuid: "fcb7dd22db7142e8a944cf470c55e43e" member_type: VOTER last_known_addr { host: "127.9.176.252" port: 46809 } } }
I20260808 22:26:23.890164 11291 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcb7dd22db7142e8a944cf470c55e43e [sys.catalog]: This master's current role is: FOLLOWER
I20260808 22:26:23.890746 11368 mvcc.cc:205] Tried to move back new op lower bound from 7316389821939752960 to 7316389821846069248. Current Snapshot: MvccSnapshot[applied={T|T < 7316389821939752960}]
I20260808 22:26:23.892398 11362 catalog_manager.cc:1347] Generated new cluster ID: c37962428e8942048518349c7bf4816e
I20260808 22:26:23.892464 11362 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:26:23.939132 11362 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:26:23.940428 11362 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:26:23.962762 11362 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f180cbc2339f4ee8abe6d13f52d13268: Generated new TSK 0
I20260808 22:26:23.963604 11362 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 22:26:23.993975 11055 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:26:23.990878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53128 (local address 127.9.176.254:35165)
0808 22:26:23.991005 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0808 22:26:23.991009 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:26:23.992849 (+ 1840us) server_negotiation.cc:407] Connection header received
0808 22:26:23.993082 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:26:23.993086 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:26:23.993138 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:26:23.993191 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:26:23.993844 (+ 653us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53128: BlockingRecv error: recv got EOF from 127.0.0.1:53128 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260808 22:26:23.995379 11287 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:26:23.991616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55986 (local address 127.9.176.252:46809)
0808 22:26:23.991750 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0808 22:26:23.991755 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:26:23.994468 (+ 2713us) server_negotiation.cc:407] Connection header received
0808 22:26:23.994654 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:26:23.994657 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:26:23.994702 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:26:23.994749 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:26:23.995255 (+ 506us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55986: BlockingRecv error: recv got EOF from 127.0.0.1:55986 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260808 22:26:23.996748 11179 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:26:23.992202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59662 (local address 127.9.176.253:42637)
0808 22:26:23.992313 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0808 22:26:23.992318 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:26:23.995850 (+ 3532us) server_negotiation.cc:407] Connection header received
0808 22:26:23.996026 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:26:23.996030 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:26:23.996082 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:26:23.996132 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:26:23.996630 (+ 498us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59662: BlockingRecv error: recv got EOF from 127.0.0.1:59662 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260808 22:26:24.003594 11179 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:26:24.001989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59668 (local address 127.9.176.253:42637)
0808 22:26:24.002095 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0808 22:26:24.002099 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:26:24.002595 (+ 496us) server_negotiation.cc:407] Connection header received
0808 22:26:24.002823 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:26:24.002827 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:26:24.002906 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:26:24.002963 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:26:24.003484 (+ 521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59668: BlockingRecv error: recv got EOF from 127.0.0.1:59668 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260808 22:26:24.008738 11179 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:26:24.007234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59682 (local address 127.9.176.253:42637)
0808 22:26:24.007336 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0808 22:26:24.007340 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:26:24.007820 (+ 480us) server_negotiation.cc:407] Connection header received
0808 22:26:24.008011 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:26:24.008014 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:26:24.008065 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:26:24.008117 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:26:24.008645 (+ 528us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59682: BlockingRecv error: recv got EOF from 127.0.0.1:59682 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260808 22:26:24.014263 11179 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:26:24.012970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59692 (local address 127.9.176.253:42637)
0808 22:26:24.013081 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0808 22:26:24.013084 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:26:24.013362 (+ 278us) server_negotiation.cc:407] Connection header received
0808 22:26:24.013567 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:26:24.013571 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:26:24.013621 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:26:24.013674 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:26:24.014175 (+ 501us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59692: BlockingRecv error: recv got EOF from 127.0.0.1:59692 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260808 22:26:24.021660 11179 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:26:24.020307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59696 (local address 127.9.176.253:42637)
0808 22:26:24.020422 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0808 22:26:24.020425 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:26:24.020848 (+ 423us) server_negotiation.cc:407] Connection header received
0808 22:26:24.021037 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:26:24.021040 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:26:24.021083 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:26:24.021128 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:26:24.021570 (+ 442us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59696: BlockingRecv error: recv got EOF from 127.0.0.1:59696 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260808 22:26:24.033416 11179 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:26:24.031847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59700 (local address 127.9.176.253:42637)
0808 22:26:24.031951 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0808 22:26:24.031954 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:26:24.032438 (+ 484us) server_negotiation.cc:407] Connection header received
0808 22:26:24.032643 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:26:24.032646 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:26:24.032702 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:26:24.032751 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:26:24.033320 (+ 569us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59700: BlockingRecv error: recv got EOF from 127.0.0.1:59700 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260808 22:26:24.052605 11179 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:26:24.051483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59716 (local address 127.9.176.253:42637)
0808 22:26:24.051603 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0808 22:26:24.051607 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:26:24.051835 (+ 228us) server_negotiation.cc:407] Connection header received
0808 22:26:24.051971 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:26:24.051973 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:26:24.052017 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:26:24.052084 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:26:24.052495 (+ 411us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59716: BlockingRecv error: recv got EOF from 127.0.0.1:59716 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260808 22:26:24.087728 11179 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:26:24.086653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59720 (local address 127.9.176.253:42637)
0808 22:26:24.086785 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0808 22:26:24.086789 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:26:24.086937 (+ 148us) server_negotiation.cc:407] Connection header received
0808 22:26:24.087064 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:26:24.087066 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:26:24.087122 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:26:24.087199 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:26:24.087634 (+ 435us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59720: BlockingRecv error: recv got EOF from 127.0.0.1:59720 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260808 22:26:24.155010 11179 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:26:24.153778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59734 (local address 127.9.176.253:42637)
0808 22:26:24.153941 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0808 22:26:24.153947 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 22:26:24.154141 (+ 194us) server_negotiation.cc:407] Connection header received
0808 22:26:24.154283 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:26:24.154286 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:26:24.154334 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:26:24.154403 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:26:24.154901 (+ 498us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59734: BlockingRecv error: recv got EOF from 127.0.0.1:59734 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260808 22:26:24.286250 11179 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:26:24.285142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59740 (local address 127.9.176.253:42637)
0808 22:26:24.285286 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0808 22:26:24.285292 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 22:26:24.285384 (+ 92us) server_negotiation.cc:407] Connection header received
0808 22:26:24.285547 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:26:24.285551 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:26:24.285599 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:26:24.285667 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:26:24.286128 (+ 461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59740: BlockingRecv error: recv got EOF from 127.0.0.1:59740 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260808 22:26:24.545511 11179 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:26:24.544282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59754 (local address 127.9.176.253:42637)
0808 22:26:24.544432 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0808 22:26:24.544436 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:26:24.544545 (+ 109us) server_negotiation.cc:407] Connection header received
0808 22:26:24.544702 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:26:24.544705 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:26:24.544786 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:26:24.544860 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:26:24.545379 (+ 519us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59754: BlockingRecv error: recv got EOF from 127.0.0.1:59754 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 22:26:24.832103 11311 catalog_manager.cc:1259] Loaded cluster ID: c37962428e8942048518349c7bf4816e
I20260808 22:26:24.832190 11311 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fcb7dd22db7142e8a944cf470c55e43e: loading cluster ID for follower catalog manager: success
I20260808 22:26:24.833994 11311 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fcb7dd22db7142e8a944cf470c55e43e: acquiring CA information for follower catalog manager: success
I20260808 22:26:24.835053 11325 catalog_manager.cc:1259] Loaded cluster ID: c37962428e8942048518349c7bf4816e
I20260808 22:26:24.835122 11325 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 81a36fa646864b2080b554566106f120: loading cluster ID for follower catalog manager: success
I20260808 22:26:24.835479 11311 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fcb7dd22db7142e8a944cf470c55e43e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 22:26:24.836694 11325 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 81a36fa646864b2080b554566106f120: acquiring CA information for follower catalog manager: success
I20260808 22:26:24.837703 11325 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 81a36fa646864b2080b554566106f120: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 22:26:25.063463 11470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:25.059513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59758 (local address 127.9.176.253:42637)
0808 22:26:25.059767 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0808 22:26:25.059772 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:26:25.059842 (+ 70us) server_negotiation.cc:407] Connection header received
0808 22:26:25.060016 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:26:25.060019 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:26:25.060149 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:26:25.060246 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:26:25.060519 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:25.061097 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:25.061960 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:25.062206 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:25.062296 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:26:25.062629 (+ 333us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:26:25.063065 (+ 436us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:26:25.063181 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0808 22:26:25.063251 (+ 70us) server_negotiation.cc:299] Negotiation successful
0808 22:26:25.063311 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":82,"threads_started":1}
I20260808 22:26:25.069725 11480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:25.066387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55998 (local address 127.9.176.252:46809)
0808 22:26:25.066660 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0808 22:26:25.066663 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:26:25.066677 (+ 14us) server_negotiation.cc:407] Connection header received
0808 22:26:25.066761 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:26:25.066763 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:26:25.066876 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:26:25.066984 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:26:25.067148 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:25.067658 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:25.068466 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:25.068687 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:25.068751 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:26:25.069005 (+ 254us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:26:25.069407 (+ 402us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:26:25.069493 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0808 22:26:25.069556 (+ 63us) server_negotiation.cc:299] Negotiation successful
0808 22:26:25.069602 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":74,"threads_started":1}
I20260808 22:26:25.076444 11480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:25.072806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56002 (local address 127.9.176.252:46809)
0808 22:26:25.072930 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0808 22:26:25.072934 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:26:25.073321 (+ 387us) server_negotiation.cc:407] Connection header received
0808 22:26:25.073361 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:26:25.073363 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:26:25.073404 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:26:25.073478 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:26:25.073701 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:25.074386 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:25.075264 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:25.075581 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:25.075675 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:26:25.075925 (+ 250us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:26:25.076006 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:26:25.076073 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0808 22:26:25.076260 (+ 187us) server_negotiation.cc:299] Negotiation successful
0808 22:26:25.076317 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260808 22:26:25.076611 11470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:25.072430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59760 (local address 127.9.176.253:42637)
0808 22:26:25.072556 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0808 22:26:25.072560 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:26:25.072909 (+ 349us) server_negotiation.cc:407] Connection header received
0808 22:26:25.073073 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:26:25.073077 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:26:25.073129 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:26:25.073202 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:26:25.073434 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:25.074156 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:25.075336 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:25.075656 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:25.075744 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:26:25.075979 (+ 235us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:26:25.076051 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:26:25.076404 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0808 22:26:25.076438 (+ 34us) server_negotiation.cc:299] Negotiation successful
0808 22:26:25.076482 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 22:26:25.077206 11490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:25.072430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53136 (local address 127.9.176.254:35165)
0808 22:26:25.072688 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0808 22:26:25.072692 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:26:25.072884 (+ 192us) server_negotiation.cc:407] Connection header received
0808 22:26:25.073004 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:26:25.073007 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:26:25.073125 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:26:25.073231 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:26:25.073561 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:25.074125 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:25.075757 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:25.076078 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:25.076166 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:26:25.076479 (+ 313us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:26:25.076807 (+ 328us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:26:25.076926 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0808 22:26:25.077020 (+ 94us) server_negotiation.cc:299] Negotiation successful
0808 22:26:25.077077 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":85,"threads_started":1}
I20260808 22:26:25.080006 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 10941
I20260808 22:26:25.091049 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 11064
I20260808 22:26:25.104409 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 11180
2026-08-08T22:26:25Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3158 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786227706282457-9923-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 22:26:25 dist-test-slave-ncp9 krb5kdc[11495](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 22:26:25 dist-test-slave-ncp9 krb5kdc[11495](info): set up 2 sockets
Aug 08 22:26:25 dist-test-slave-ncp9 krb5kdc[11495](info): commencing operation
krb5kdc: starting...
W20260808 22:26:26.236634 9923 mini_kdc.cc:121] Time spent starting KDC: real 1.100s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786227706282457-9923-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786227706282457-9923-0/krb5kdc/test-user.keytab.
Aug 08 22:26:26 dist-test-slave-ncp9 krb5kdc[11495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T22:26:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:26:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.176.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.176.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.176.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab.
Entry for principal kudu/127.9.176.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab.
I20260808 22:26:26.353411 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786227706282457-9923-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786227706282457-9923-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786227706282457-9923-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:40175
--webserver_interface=127.9.176.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab
--principal=kudu/127.9.176.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:33709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.176.254:40175,127.9.176.253:37351,127.9.176.252:37405
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:26:26.467916 11520 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 22:26:26.468292 11520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:26:26.468343 11520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:26:26.468405 11520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:26:26.473660 11520 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:26:26.473714 11520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:26:26.473732 11520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:26:26.473748 11520 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:26:26.473770 11520 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:26:26.478873 11520 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:33709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786227706282457-9923-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786227706282457-9923-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.176.254:40175,127.9.176.253:37351,127.9.176.252:37405
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.254.keytab
--principal=kudu/127.9.176.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.254:40175
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.176.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11520
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786227706282457-9923-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:26:26.479822 11520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:26:26.480757 11520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:26:26.488801 11532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:26:26.488834 11520 server_base.cc:1034] running on GCE node
W20260808 22:26:26.488790 11529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:26:26.488790 11530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:26:26.489678 11520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:26:26.490729 11520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:26:26.491881 11520 hybrid_clock.cc:648] HybridClock initialized: now 1786227986491866 us; error 41 us; skew 500 ppm
Aug 08 22:26:26 dist-test-slave-ncp9 krb5kdc[11495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227986, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:26:26.494580 11520 init.cc:377] Logged in from keytab as kudu/127.9.176.254@KRBTEST.COM (short username kudu)
I20260808 22:26:26.495456 11520 webserver.cc:460] Webserver started at http://127.9.176.254:33779/ using document root <none> and password file <none>
I20260808 22:26:26.495982 11520 fs_manager.cc:359] Metadata directory not provided
I20260808 22:26:26.496062 11520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:26:26.496271 11520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:26:26.498345 11520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786227706282457-9923-0/minicluster-data/master-0/data/instance:
uuid: "bdcf35097fa34f358c646a95b3f82c99"
format_stamp: "Formatted at 2026-08-08 22:26:26 on dist-test-slave-ncp9"
server_key: "ee4adbfd03f14c240dcbef12c8938fd5"
server_key_iv: "e1f806473d37fd6d8dbe5d25ae9de225"
server_key_version: "encryptionkey@0"
I20260808 22:26:26.498894 11520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786227706282457-9923-0/minicluster-data/master-0/wal/instance:
uuid: "bdcf35097fa34f358c646a95b3f82c99"
format_stamp: "Formatted at 2026-08-08 22:26:26 on dist-test-slave-ncp9"
server_key: "ee4adbfd03f14c240dcbef12c8938fd5"
server_key_iv: "e1f806473d37fd6d8dbe5d25ae9de225"
server_key_version: "encryptionkey@0"
I20260808 22:26:26.502717 11520 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 22:26:26.505149 11543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:26:26.506438 11520 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 22:26:26.506557 11520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786227706282457-9923-0/minicluster-data/master-0/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786227706282457-9923-0/minicluster-data/master-0/wal
uuid: "bdcf35097fa34f358c646a95b3f82c99"
format_stamp: "Formatted at 2026-08-08 22:26:26 on dist-test-slave-ncp9"
server_key: "ee4adbfd03f14c240dcbef12c8938fd5"
server_key_iv: "e1f806473d37fd6d8dbe5d25ae9de225"
server_key_version: "encryptionkey@0"
I20260808 22:26:26.506659 11520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786227706282457-9923-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786227706282457-9923-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786227706282457-9923-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:26:26.531955 11520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:26:26.535111 11520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:26:26.535287 11520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:26:26.542290 11520 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.254:40175
I20260808 22:26:26.542554 11616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.254:40175 every 8 connection(s)
I20260808 22:26:26.543283 11520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786227706282457-9923-0/minicluster-data/master-0/data/info.pb
I20260808 22:26:26.546031 11617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:26:26.549503 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 11520
I20260808 22:26:26.549640 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786227706282457-9923-0/minicluster-data/master-0/wal/instance
I20260808 22:26:26.549966 9923 external_mini_cluster.cc:1428] Setting key c460f1d729db660e27e1c538e2b9a5ff
I20260808 22:26:26.549860 11617 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.176.254" port: 40175 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.9.176.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.176.253@KRBTEST.COM" created.
Aug 08 22:26:26 dist-test-slave-ncp9 krb5kdc[11495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227986, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.254@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
I20260808 22:26:26.566586 11621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:26.552454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:40175 (local address 127.0.0.1:39330)
0808 22:26:26.552985 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0808 22:26:26.553029 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 22:26:26.553875 (+ 846us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:26:26.554539 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:26:26.554544 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:26:26.554946 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:26:26.555185 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:26:26.555198 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:26:26.556323 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:26:26.556326 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:26:26.557105 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:26:26.557114 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:26:26.557703 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:26:26.557707 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:26:26.557838 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:26:26.558357 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:26:26.558373 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:26:26.561754 (+ 3381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:26:26.563807 (+ 2053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:26:26.563815 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:26:26.563823 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:26:26.564042 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:26:26.564396 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:26:26.564398 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:26:26.564400 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:26:26.564496 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:26:26.565252 (+ 756us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:26:26.565257 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:26:26.565383 (+ 126us) client_negotiation.cc:769] Sending connection context
0808 22:26:26.565511 (+ 128us) client_negotiation.cc:240] Negotiation successful
0808 22:26:26.565776 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"mutex_wait_us":3,"thread_start_us":127,"threads_started":1}
I20260808 22:26:26.566589 11623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:26.553396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39330 (local address 127.9.176.254:40175)
0808 22:26:26.553682 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0808 22:26:26.553693 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 22:26:26.553735 (+ 42us) server_negotiation.cc:407] Connection header received
0808 22:26:26.554154 (+ 419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:26:26.554158 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:26:26.554337 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:26:26.554494 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:26:26.555321 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:26.556210 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:26.557207 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:26.557568 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:26.557716 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:26:26.561900 (+ 4184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:26:26.561905 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:26:26.561915 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:26:26.561938 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:26:26.563701 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:26:26.564168 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:26:26.564174 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:26:26.564175 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:26:26.564312 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:26:26.564587 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:26:26.564590 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:26:26.564592 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:26:26.565144 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:26:26.565286 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0808 22:26:26.565637 (+ 351us) server_negotiation.cc:299] Negotiation successful
0808 22:26:26.565852 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":87,"server-negotiator.queue_time_us":189,"spinlock_wait_cycles":1920,"thread_start_us":85,"threads_started":1}
I20260808 22:26:26.569077 11617 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.176.253" port: 37351 } has no permanent_uuid. Determining permanent_uuid...
I20260808 22:26:26.569705 11621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:26.569484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.253:37351 (local address 127.0.0.1:40600)
0808 22:26:26.569588 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0808 22:26:26.569654 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.253:37351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260808 22:26:26.569912 11546 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.176.253:37351: connect: Connection refused (error 111)
I20260808 22:26:26.570672 11621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:26.570473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.253:37351 (local address 127.0.0.1:40614)
0808 22:26:26.570571 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0808 22:26:26.570624 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.253:37351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260808 22:26:26.570804 11617 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.176.253:37351: Network error: Client connection negotiation failed: client connection to 127.9.176.253:37351: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.176.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.253.keytab.
Entry for principal kudu/127.9.176.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.253.keytab.
I20260808 22:26:26.575941 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786227706282457-9923-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786227706282457-9923-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786227706282457-9923-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786227706282457-9923-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.253:37351
--webserver_interface=127.9.176.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.253.keytab
--principal=kudu/127.9.176.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:33709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.176.254:40175,127.9.176.253:37351,127.9.176.252:37405
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 22:26:26.620052 11617 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.176.253" port: 37351 } attempt: 1
I20260808 22:26:26.620956 11621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:26.620588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.253:37351 (local address 127.0.0.1:40626)
0808 22:26:26.620750 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0808 22:26:26.620825 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.253:37351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260808 22:26:26.621723 11621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:26.621500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.253:37351 (local address 127.0.0.1:40634)
0808 22:26:26.621605 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0808 22:26:26.621659 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.253:37351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260808 22:26:26.621868 11617 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.176.253:37351: Network error: Client connection negotiation failed: client connection to 127.9.176.253:37351: connect: Connection refused (error 111)
W20260808 22:26:26.684533 11627 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 22:26:26.684808 11627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:26:26.684830 11627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:26:26.684867 11627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:26:26.688993 11627 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:26:26.689064 11627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:26:26.689081 11627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:26:26.689097 11627 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:26:26.689111 11627 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:26:26.690032 11617 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.176.253" port: 37351 } attempt: 2
I20260808 22:26:26.690881 11621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:26.690549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.253:37351 (local address 127.0.0.1:40650)
0808 22:26:26.690705 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0808 22:26:26.690774 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.253:37351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260808 22:26:26.691592 11621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:26.691379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.253:37351 (local address 127.0.0.1:40660)
0808 22:26:26.691482 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0808 22:26:26.691531 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.253:37351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260808 22:26:26.691707 11617 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.176.253:37351: Network error: Client connection negotiation failed: client connection to 127.9.176.253:37351: connect: Connection refused (error 111)
I20260808 22:26:26.694653 11627 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:33709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786227706282457-9923-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786227706282457-9923-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.9.176.254:40175,127.9.176.253:37351,127.9.176.252:37405
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.253.keytab
--principal=kudu/127.9.176.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.253:37351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786227706282457-9923-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.9.176.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11627
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786227706282457-9923-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:26:26.695674 11627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:26:26.696668 11627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:26:26.705156 11638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:26:26.705191 11639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:26:26.705156 11641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:26:26.705567 11627 server_base.cc:1034] running on GCE node
I20260808 22:26:26.706095 11627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:26:26.707155 11627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:26:26.708321 11627 hybrid_clock.cc:648] HybridClock initialized: now 1786227986708306 us; error 26 us; skew 500 ppm
Aug 08 22:26:26 dist-test-slave-ncp9 krb5kdc[11495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227986, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:26:26.711158 11627 init.cc:377] Logged in from keytab as kudu/127.9.176.253@KRBTEST.COM (short username kudu)
I20260808 22:26:26.712069 11627 webserver.cc:460] Webserver started at http://127.9.176.253:40701/ using document root <none> and password file <none>
I20260808 22:26:26.712592 11627 fs_manager.cc:359] Metadata directory not provided
I20260808 22:26:26.712651 11627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:26:26.712817 11627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:26:26.714890 11627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786227706282457-9923-0/minicluster-data/master-1/data/instance:
uuid: "b3a4e460127749249a2d1e8098507182"
format_stamp: "Formatted at 2026-08-08 22:26:26 on dist-test-slave-ncp9"
server_key: "0cc66c05522410f3956dbc64da2edab4"
server_key_iv: "c358b7326d597f3dfc0886020522919d"
server_key_version: "encryptionkey@0"
I20260808 22:26:26.715408 11627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786227706282457-9923-0/minicluster-data/master-1/wal/instance:
uuid: "b3a4e460127749249a2d1e8098507182"
format_stamp: "Formatted at 2026-08-08 22:26:26 on dist-test-slave-ncp9"
server_key: "0cc66c05522410f3956dbc64da2edab4"
server_key_iv: "c358b7326d597f3dfc0886020522919d"
server_key_version: "encryptionkey@0"
I20260808 22:26:26.719132 11627 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 22:26:26.721503 11650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:26:26.722822 11627 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 22:26:26.722949 11627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786227706282457-9923-0/minicluster-data/master-1/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786227706282457-9923-0/minicluster-data/master-1/wal
uuid: "b3a4e460127749249a2d1e8098507182"
format_stamp: "Formatted at 2026-08-08 22:26:26 on dist-test-slave-ncp9"
server_key: "0cc66c05522410f3956dbc64da2edab4"
server_key_iv: "c358b7326d597f3dfc0886020522919d"
server_key_version: "encryptionkey@0"
I20260808 22:26:26.723050 11627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786227706282457-9923-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786227706282457-9923-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786227706282457-9923-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:26:26.743582 11627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:26:26.746760 11627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:26:26.746951 11627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:26:26.754076 11627 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.253:37351
I20260808 22:26:26.754320 11719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.253:37351 every 8 connection(s)
I20260808 22:26:26.755045 11627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786227706282457-9923-0/minicluster-data/master-1/data/info.pb
I20260808 22:26:26.757858 11721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:26:26.761827 11721 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.176.254" port: 40175 } has no permanent_uuid. Determining permanent_uuid...
I20260808 22:26:26.762154 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 11627
I20260808 22:26:26.762302 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786227706282457-9923-0/minicluster-data/master-1/wal/instance
I20260808 22:26:26.762624 9923 external_mini_cluster.cc:1428] Setting key 26ec462f780e3ad9bf47964ef004f09e
WARNING: no policy specified for kudu/127.9.176.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.176.252@KRBTEST.COM" created.
Aug 08 22:26:26 dist-test-slave-ncp9 krb5kdc[11495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227986, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.253@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
I20260808 22:26:26.777827 11623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:26.763833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39334 (local address 127.9.176.254:40175)
0808 22:26:26.763979 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0808 22:26:26.763983 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:26:26.765126 (+ 1143us) server_negotiation.cc:407] Connection header received
0808 22:26:26.766102 (+ 976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:26:26.766106 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:26:26.766167 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:26:26.766241 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:26:26.767524 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:26.768068 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:26.769310 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:26.769550 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:26.769645 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:26:26.774266 (+ 4621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:26:26.774270 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:26:26.774272 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:26:26.774299 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:26:26.775708 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:26:26.776228 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:26:26.776231 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:26:26.776232 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:26:26.776314 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:26:26.776666 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:26:26.776669 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:26:26.776670 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:26:26.776861 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:26:26.776985 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0808 22:26:26.777521 (+ 536us) server_negotiation.cc:299] Negotiation successful
0808 22:26:26.777663 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 22:26:26.778613 11728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:26.764266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:40175 (local address 127.0.0.1:39334)
0808 22:26:26.764970 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0808 22:26:26.765021 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0808 22:26:26.765884 (+ 863us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:26:26.766423 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:26:26.766430 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:26:26.766810 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:26:26.767369 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:26:26.767390 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:26:26.768195 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:26:26.768197 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:26:26.769185 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:26:26.769197 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:26:26.769699 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:26:26.769704 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:26:26.769875 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:26:26.770420 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:26:26.770436 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:26:26.774077 (+ 3641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:26:26.775829 (+ 1752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:26:26.775835 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:26:26.775849 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:26:26.776119 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:26:26.776400 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:26:26.776403 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:26:26.776405 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:26:26.776562 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:26:26.776983 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:26:26.776990 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:26:26.777263 (+ 273us) client_negotiation.cc:769] Sending connection context
0808 22:26:26.777427 (+ 164us) client_negotiation.cc:240] Negotiation successful
0808 22:26:26.777761 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":207,"threads_started":1}
I20260808 22:26:26.780129 11721 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.176.253" port: 37351 } has no permanent_uuid. Determining permanent_uuid...
I20260808 22:26:26.782843 11617 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.176.253" port: 37351 } attempt: 3
Aug 08 22:26:26 dist-test-slave-ncp9 krb5kdc[11495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227986, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.253@KRBTEST.COM for kudu/127.9.176.253@KRBTEST.COM
Aug 08 22:26:26 dist-test-slave-ncp9 krb5kdc[11495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227986, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.254@KRBTEST.COM for kudu/127.9.176.253@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.176.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.252.keytab.
Entry for principal kudu/127.9.176.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.252.keytab.
I20260808 22:26:26.789105 9923 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
/tmp/dist-test-taskqqDyFd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786227706282457-9923-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786227706282457-9923-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786227706282457-9923-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786227706282457-9923-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.252:37405
--webserver_interface=127.9.176.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.252.keytab
--principal=kudu/127.9.176.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.176.212:33709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.176.254:40175,127.9.176.253:37351,127.9.176.252:37405
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786227706282457-9923-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786227706282457-9923-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786227706282457-9923-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 22:26:26.790194 11730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:26.780763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40668 (local address 127.9.176.253:37351)
0808 22:26:26.781068 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0808 22:26:26.781078 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 22:26:26.781097 (+ 19us) server_negotiation.cc:407] Connection header received
0808 22:26:26.781133 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:26:26.781137 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:26:26.781288 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:26:26.781402 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:26:26.781726 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:26.782550 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:26.783480 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:26.784002 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:26.784111 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:26:26.786405 (+ 2294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:26:26.786416 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:26:26.786430 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:26:26.786455 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:26:26.788133 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:26:26.788575 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:26:26.788581 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:26:26.788583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:26:26.788664 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:26:26.789150 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:26:26.789153 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:26:26.789155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:26:26.789453 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:26:26.789584 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0808 22:26:26.789840 (+ 256us) server_negotiation.cc:299] Negotiation successful
0808 22:26:26.789977 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":100,"threads_started":1}
I20260808 22:26:26.790220 11728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:26.780648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.253:37351 (local address 127.0.0.1:40668)
0808 22:26:26.780817 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0808 22:26:26.780830 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 22:26:26.780922 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:26:26.781415 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:26:26.781418 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:26:26.781550 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:26:26.781637 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:26:26.781642 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:26:26.782657 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:26:26.782660 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:26:26.783356 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:26:26.783367 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:26:26.784145 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:26:26.784149 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:26:26.784249 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:26:26.784720 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:26:26.784734 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:26:26.786229 (+ 1495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:26:26.788230 (+ 2001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:26:26.788233 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:26:26.788235 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:26:26.788469 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:26:26.788968 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:26:26.788972 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:26:26.788974 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:26:26.789065 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:26:26.789551 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:26:26.789554 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:26:26.789638 (+ 84us) client_negotiation.cc:769] Sending connection context
0808 22:26:26.789723 (+ 85us) client_negotiation.cc:240] Negotiation successful
0808 22:26:26.789846 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102}
I20260808 22:26:26.792281 11621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:26.783222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.253:37351 (local address 127.0.0.1:40676)
0808 22:26:26.783338 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0808 22:26:26.783352 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 22:26:26.783417 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:26:26.783786 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:26:26.783789 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:26:26.783977 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:26:26.784103 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:26:26.784111 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:26:26.785141 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:26:26.785144 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:26:26.785930 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:26:26.785941 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:26:26.786490 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:26:26.786493 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:26:26.786586 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:26:26.787164 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:26:26.787175 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:26:26.788341 (+ 1166us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:26:26.790844 (+ 2503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:26:26.790848 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:26:26.790850 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:26:26.791038 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:26:26.791294 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:26:26.791297 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:26:26.791299 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:26:26.791379 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:26:26.791783 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:26:26.791786 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:26:26.791888 (+ 102us) client_negotiation.cc:769] Sending connection context
0808 22:26:26.791991 (+ 103us) client_negotiation.cc:240] Negotiation successful
0808 22:26:26.792130 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260808 22:26:26.792331 11731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:26.783288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40676 (local address 127.9.176.253:37351)
0808 22:26:26.783598 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0808 22:26:26.783609 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 22:26:26.783618 (+ 9us) server_negotiation.cc:407] Connection header received
0808 22:26:26.783644 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:26:26.783645 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:26:26.783677 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:26:26.783742 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:26:26.784219 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:26.785030 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:26.786040 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:26.786382 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:26.786473 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:26:26.788497 (+ 2024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:26:26.788500 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:26:26.788502 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:26:26.788526 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:26:26.790714 (+ 2188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:26:26.791138 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:26:26.791141 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:26:26.791142 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:26:26.791206 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:26:26.791464 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:26:26.791467 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:26:26.791469 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:26:26.791680 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:26:26.791776 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0808 22:26:26.792055 (+ 279us) server_negotiation.cc:299] Negotiation successful
0808 22:26:26.792164 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":98,"threads_started":1}
I20260808 22:26:26.792789 11721 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.176.252" port: 37405 } has no permanent_uuid. Determining permanent_uuid...
I20260808 22:26:26.792999 11617 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.176.252" port: 37405 } has no permanent_uuid. Determining permanent_uuid...
I20260808 22:26:26.793572 11728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:26.793268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.252:37405 (local address 127.0.0.1:43428)
0808 22:26:26.793405 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0808 22:26:26.793505 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.252:37405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260808 22:26:26.793720 11621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:26.793463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.252:37405 (local address 127.0.0.1:43430)
0808 22:26:26.793589 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0808 22:26:26.793656 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.252:37405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260808 22:26:26.793799 11653 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.176.252:37405: connect: Connection refused (error 111)
I20260808 22:26:26.794430 11621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:26.794189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.252:37405 (local address 127.0.0.1:43434)
0808 22:26:26.794327 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0808 22:26:26.794374 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.252:37405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260808 22:26:26.794581 11617 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.176.252:37405: Network error: Client connection negotiation failed: client connection to 127.9.176.252:37405: connect: Connection refused (error 111)
I20260808 22:26:26.795051 11728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:26.794832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.252:37405 (local address 127.0.0.1:43444)
0808 22:26:26.794936 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0808 22:26:26.794986 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.252:37405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260808 22:26:26.795207 11721 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.176.252:37405: Network error: Client connection negotiation failed: client connection to 127.9.176.252:37405: connect: Connection refused (error 111)
I20260808 22:26:26.825774 11617 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.176.252" port: 37405 } attempt: 1
I20260808 22:26:26.826710 11621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:26.826370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.252:37405 (local address 127.0.0.1:43458)
0808 22:26:26.826543 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0808 22:26:26.826622 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.252:37405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260808 22:26:26.827490 11621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:26.827260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.252:37405 (local address 127.0.0.1:43474)
0808 22:26:26.827370 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0808 22:26:26.827427 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.252:37405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260808 22:26:26.827631 11617 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.176.252:37405: Network error: Client connection negotiation failed: client connection to 127.9.176.252:37405: connect: Connection refused (error 111)
I20260808 22:26:26.844455 11721 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.176.252" port: 37405 } attempt: 1
I20260808 22:26:26.845288 11728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:26.845006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.252:37405 (local address 127.0.0.1:43486)
0808 22:26:26.845147 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0808 22:26:26.845209 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.252:37405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260808 22:26:26.846179 11728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:26.845945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.252:37405 (local address 127.0.0.1:43502)
0808 22:26:26.846072 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0808 22:26:26.846120 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.252:37405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260808 22:26:26.846328 11721 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.176.252:37405: Network error: Client connection negotiation failed: client connection to 127.9.176.252:37405: connect: Connection refused (error 111)
I20260808 22:26:26.902791 11617 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.176.252" port: 37405 } attempt: 2
I20260808 22:26:26.903606 11621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:26.903314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.252:37405 (local address 127.0.0.1:43518)
0808 22:26:26.903459 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0808 22:26:26.903530 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.252:37405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260808 22:26:26.904292 11621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:26.904090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.252:37405 (local address 127.0.0.1:43532)
0808 22:26:26.904193 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0808 22:26:26.904237 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.252:37405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260808 22:26:26.904233 11733 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 22:26:26.904422 11617 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.176.252:37405: Network error: Client connection negotiation failed: client connection to 127.9.176.252:37405: connect: Connection refused (error 111)
W20260808 22:26:26.904534 11733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:26:26.904574 11733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:26:26.904634 11733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:26:26.908565 11733 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:26:26.908623 11733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:26:26.908649 11733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:26:26.908672 11733 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:26:26.908702 11733 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:26:26.914040 11733 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.176.212:33709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786227706282457-9923-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786227706282457-9923-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.9.176.254:40175,127.9.176.253:37351,127.9.176.252:37405
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786227706282457-9923-0/krb5kdc/kudu_127.9.176.252.keytab
--principal=kudu/127.9.176.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.176.252:37405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786227706282457-9923-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.9.176.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11733
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786227706282457-9923-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 21:47:48 UTC on ccycloud.quasar-pjuikp.root.comops.site
build id 4575493
I20260808 22:26:26.914475 11721 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.176.252" port: 37405 } attempt: 2
I20260808 22:26:26.915211 11733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:26:26.915234 11728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:26.914964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.252:37405 (local address 127.0.0.1:43548)
0808 22:26:26.915100 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0808 22:26:26.915162 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.252:37405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260808 22:26:26.915985 11728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:26.915778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.252:37405 (local address 127.0.0.1:43554)
0808 22:26:26.915888 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0808 22:26:26.915932 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.176.252:37405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260808 22:26:26.916136 11721 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.176.252:37405: Network error: Client connection negotiation failed: client connection to 127.9.176.252:37405: connect: Connection refused (error 111)
I20260808 22:26:26.916368 11733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:26:26.924546 11739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:26:26.924571 11748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:26:26.924551 11741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:26:26.924942 11733 server_base.cc:1034] running on GCE node
I20260808 22:26:26.925500 11733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:26:26.926556 11733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:26:26.927721 11733 hybrid_clock.cc:648] HybridClock initialized: now 1786227986927699 us; error 32 us; skew 500 ppm
Aug 08 22:26:26 dist-test-slave-ncp9 krb5kdc[11495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227986, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:26:26.930346 11733 init.cc:377] Logged in from keytab as kudu/127.9.176.252@KRBTEST.COM (short username kudu)
I20260808 22:26:26.931232 11733 webserver.cc:460] Webserver started at http://127.9.176.252:35833/ using document root <none> and password file <none>
I20260808 22:26:26.931769 11733 fs_manager.cc:359] Metadata directory not provided
I20260808 22:26:26.931835 11733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:26:26.932036 11733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:26:26.934139 11733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786227706282457-9923-0/minicluster-data/master-2/data/instance:
uuid: "5c095aa9f0f54125ad918ba8ebaa6be4"
format_stamp: "Formatted at 2026-08-08 22:26:26 on dist-test-slave-ncp9"
server_key: "fe414209840421bdfdac98015f089bd0"
server_key_iv: "2f01f9f275fae8f4fd741ca95d46002c"
server_key_version: "encryptionkey@0"
I20260808 22:26:26.934736 11733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786227706282457-9923-0/minicluster-data/master-2/wal/instance:
uuid: "5c095aa9f0f54125ad918ba8ebaa6be4"
format_stamp: "Formatted at 2026-08-08 22:26:26 on dist-test-slave-ncp9"
server_key: "fe414209840421bdfdac98015f089bd0"
server_key_iv: "2f01f9f275fae8f4fd741ca95d46002c"
server_key_version: "encryptionkey@0"
I20260808 22:26:26.938403 11733 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 22:26:26.940726 11759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:26:26.941893 11733 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 22:26:26.942008 11733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786227706282457-9923-0/minicluster-data/master-2/data,/tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786227706282457-9923-0/minicluster-data/master-2/wal
uuid: "5c095aa9f0f54125ad918ba8ebaa6be4"
format_stamp: "Formatted at 2026-08-08 22:26:26 on dist-test-slave-ncp9"
server_key: "fe414209840421bdfdac98015f089bd0"
server_key_iv: "2f01f9f275fae8f4fd741ca95d46002c"
server_key_version: "encryptionkey@0"
I20260808 22:26:26.942102 11733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786227706282457-9923-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786227706282457-9923-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786227706282457-9923-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:26:26.965902 11733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:26:26.968744 11733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:26:26.968950 11733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:26:26.975744 11733 rpc_server.cc:307] RPC server started. Bound to: 127.9.176.252:37405
I20260808 22:26:26.976012 11831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.176.252:37405 every 8 connection(s)
I20260808 22:26:26.976697 11733 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786227706282457-9923-0/minicluster-data/master-2/data/info.pb
I20260808 22:26:26.977726 9923 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu as pid 11733
I20260808 22:26:26.977865 9923 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqqDyFd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786227706282457-9923-0/minicluster-data/master-2/wal/instance
I20260808 22:26:26.978157 9923 external_mini_cluster.cc:1428] Setting key d46b6823ae2e0b97d786b22b7522b1fa
I20260808 22:26:26.979642 11833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:26:26.984335 11833 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.176.254" port: 40175 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 22:26:26 dist-test-slave-ncp9 krb5kdc[11495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.176.252@KRBTEST.COM
Aug 08 22:26:26 dist-test-slave-ncp9 krb5kdc[11495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227986, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.252@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
I20260808 22:26:26.992456 11839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:26.979432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43568 (local address 127.9.176.252:37405)
0808 22:26:26.979852 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0808 22:26:26.979870 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0808 22:26:26.979913 (+ 43us) server_negotiation.cc:407] Connection header received
0808 22:26:26.980754 (+ 841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:26:26.980778 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:26:26.981231 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:26:26.981577 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:26:26.982236 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:26.983750 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:26.984565 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:26.985277 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:26.985436 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:26:26.987930 (+ 2494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:26:26.987936 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:26:26.987948 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:26:26.987988 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:26:26.989737 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:26:26.990145 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:26:26.990151 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:26:26.990157 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:26:26.990292 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:26:26.990547 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:26:26.990550 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:26:26.990552 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:26:26.991004 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:26:26.991153 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0808 22:26:26.991442 (+ 289us) server_negotiation.cc:299] Negotiation successful
0808 22:26:26.991770 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":119,"threads_started":1}
I20260808 22:26:26.995708 11623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:26.986293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39338 (local address 127.9.176.254:40175)
0808 22:26:26.986406 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0808 22:26:26.986409 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:26:26.986753 (+ 344us) server_negotiation.cc:407] Connection header received
0808 22:26:26.986870 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:26:26.986873 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:26:26.986923 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:26:26.986990 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:26:26.987662 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:26.988314 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:26.989489 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:26.989702 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:26.989785 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:26:26.991953 (+ 2168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:26:26.991955 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:26:26.991957 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:26:26.991981 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:26:26.993814 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:26:26.994290 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:26:26.994294 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:26:26.994296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:26:26.994386 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:26:26.994837 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:26:26.994841 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:26:26.994842 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:26:26.995052 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:26:26.995151 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0808 22:26:26.995467 (+ 316us) server_negotiation.cc:299] Negotiation successful
0808 22:26:26.995594 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260808 22:26:26.995704 11840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:26.986267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.254:40175 (local address 127.0.0.1:39338)
0808 22:26:26.986633 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0808 22:26:26.986672 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 22:26:26.986781 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:26:26.987057 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:26:26.987063 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:26:26.987384 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:26:26.987547 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:26:26.987554 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:26:26.988440 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:26:26.988443 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:26:26.989373 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:26:26.989384 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:26:26.989803 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:26:26.989807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:26:26.989913 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:26:26.990320 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:26:26.990334 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:26:26.991829 (+ 1495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:26:26.993917 (+ 2088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:26:26.993927 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:26:26.993932 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:26:26.994208 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:26:26.994670 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:26:26.994672 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:26:26.994673 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:26:26.994747 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:26:26.995169 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:26:26.995174 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:26:26.995268 (+ 94us) client_negotiation.cc:769] Sending connection context
0808 22:26:26.995373 (+ 105us) client_negotiation.cc:240] Negotiation successful
0808 22:26:26.995505 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":156,"threads_started":1}
I20260808 22:26:26.996793 11833 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.176.253" port: 37351 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 22:26:26 dist-test-slave-ncp9 krb5kdc[11495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.176.254@KRBTEST.COM
Aug 08 22:26:26 dist-test-slave-ncp9 krb5kdc[11495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227986, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.252@KRBTEST.COM for kudu/127.9.176.253@KRBTEST.COM
I20260808 22:26:27.002826 11842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:26.994520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39354 (local address 127.9.176.254:40175)
0808 22:26:26.994775 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0808 22:26:26.994781 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 22:26:26.994792 (+ 11us) server_negotiation.cc:407] Connection header received
0808 22:26:26.994819 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:26:26.994821 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:26:26.994853 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:26:26.994919 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:26:26.995573 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:26.996176 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:26.996994 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:26.997255 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:26.997349 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:26:26.999741 (+ 2392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:26:26.999744 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:26:26.999746 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:26:26.999767 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:26:27.001286 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:26:27.001687 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:26:27.001689 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:26:27.001690 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:26:27.001763 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:26:27.002007 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:26:27.002010 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:26:27.002012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:26:27.002201 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:26:27.002361 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0808 22:26:27.002553 (+ 192us) server_negotiation.cc:299] Negotiation successful
0808 22:26:27.002653 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":85,"threads_started":1}
I20260808 22:26:27.003564 11617 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.176.252" port: 37405 } attempt: 3
I20260808 22:26:27.004870 11840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:26.997254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.253:37351 (local address 127.0.0.1:40692)
0808 22:26:26.997389 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0808 22:26:26.997398 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 22:26:26.997453 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:26:26.997739 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:26:26.997741 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:26:26.997879 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:26:26.998002 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:26:26.998010 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:26:26.998732 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:26:26.998736 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:26:26.999248 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:26:26.999258 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:26:26.999732 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:26:26.999735 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:26:26.999833 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:26:27.000261 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:26:27.000272 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:26:27.001476 (+ 1204us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:26:27.003260 (+ 1784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:26:27.003263 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:26:27.003265 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:26:27.003504 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:26:27.003807 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:26:27.003810 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:26:27.003812 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:26:27.003901 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:26:27.004385 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:26:27.004387 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:26:27.004480 (+ 93us) client_negotiation.cc:769] Sending connection context
0808 22:26:27.004586 (+ 106us) client_negotiation.cc:240] Negotiation successful
0808 22:26:27.004710 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260808 22:26:27.005470 11731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:26.997405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40692 (local address 127.9.176.253:37351)
0808 22:26:26.997554 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0808 22:26:26.997559 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:26:26.997571 (+ 12us) server_negotiation.cc:407] Connection header received
0808 22:26:26.997608 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:26:26.997610 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:26:26.997648 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:26:26.997722 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:26:26.998124 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:26.998634 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:26.999358 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:26.999624 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:26.999716 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:26:27.001604 (+ 1888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:26:27.001608 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:26:27.001610 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:26:27.001638 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:26:27.003168 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:26:27.003626 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:26:27.003629 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:26:27.003630 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:26:27.003722 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:26:27.004028 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:26:27.004031 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:26:27.004033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:26:27.004253 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:26:27.004319 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0808 22:26:27.004805 (+ 486us) server_negotiation.cc:299] Negotiation successful
0808 22:26:27.005315 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 22:26:27.006515 11833 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.176.252" port: 37405 } has no permanent_uuid. Determining permanent_uuid...
I20260808 22:26:27.007278 11721 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.176.252" port: 37405 } attempt: 3
Aug 08 22:26:27 dist-test-slave-ncp9 krb5kdc[11495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227986, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.254@KRBTEST.COM for kudu/127.9.176.252@KRBTEST.COM
Aug 08 22:26:27 dist-test-slave-ncp9 krb5kdc[11495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.176.253@KRBTEST.COM
I20260808 22:26:27.012598 11621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:27.003911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.252:37405 (local address 127.0.0.1:43572)
0808 22:26:27.004074 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0808 22:26:27.004087 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 22:26:27.004167 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:26:27.004490 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:26:27.004493 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:26:27.004656 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:26:27.004745 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:26:27.004752 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:26:27.005786 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:26:27.005789 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:26:27.006378 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:26:27.006387 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:26:27.006966 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:26:27.006970 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:26:27.007065 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:26:27.007482 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:26:27.007496 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:26:27.008786 (+ 1290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:26:27.011043 (+ 2257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:26:27.011047 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:26:27.011049 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:26:27.011282 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:26:27.011581 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:26:27.011584 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:26:27.011586 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:26:27.011670 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:26:27.012119 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:26:27.012122 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:26:27.012213 (+ 91us) client_negotiation.cc:769] Sending connection context
0808 22:26:27.012306 (+ 93us) client_negotiation.cc:240] Negotiation successful
0808 22:26:27.012436 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89}
I20260808 22:26:27.013080 11839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:27.003963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43572 (local address 127.9.176.252:37405)
0808 22:26:27.004298 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0808 22:26:27.004302 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:26:27.004313 (+ 11us) server_negotiation.cc:407] Connection header received
0808 22:26:27.004341 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:26:27.004344 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:26:27.004378 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:26:27.004848 (+ 470us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:26:27.004958 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:27.005651 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:27.006478 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:27.006806 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:27.007263 (+ 457us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:26:27.008962 (+ 1699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:26:27.008966 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:26:27.008968 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:26:27.008992 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:26:27.010906 (+ 1914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:26:27.011391 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:26:27.011394 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:26:27.011396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:26:27.011478 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:26:27.011768 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:26:27.011771 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:26:27.011773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:26:27.011983 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:26:27.012074 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0808 22:26:27.012787 (+ 713us) server_negotiation.cc:299] Negotiation successful
0808 22:26:27.012927 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268}
Aug 08 22:26:27 dist-test-slave-ncp9 krb5kdc[11495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227986, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.253@KRBTEST.COM for kudu/127.9.176.252@KRBTEST.COM
I20260808 22:26:27.013577 11730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:27.004120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40708 (local address 127.9.176.253:37351)
0808 22:26:27.004235 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0808 22:26:27.004239 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:26:27.004252 (+ 13us) server_negotiation.cc:407] Connection header received
0808 22:26:27.004284 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:26:27.004287 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:26:27.004328 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:26:27.004402 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:26:27.005296 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:27.005951 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:27.006742 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:27.007200 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:27.007319 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:26:27.009936 (+ 2617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:26:27.009940 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:26:27.009943 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:26:27.009970 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:26:27.011667 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:26:27.012123 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:26:27.012126 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:26:27.012128 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:26:27.012213 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:26:27.012531 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:26:27.012535 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:26:27.012536 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:26:27.012735 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:26:27.012949 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0808 22:26:27.013309 (+ 360us) server_negotiation.cc:299] Negotiation successful
0808 22:26:27.013428 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
Aug 08 22:26:27 dist-test-slave-ncp9 krb5kdc[11495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786227986, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.176.252@KRBTEST.COM for kudu/127.9.176.252@KRBTEST.COM
I20260808 22:26:27.016546 11617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bdcf35097fa34f358c646a95b3f82c99: Bootstrap starting.
I20260808 22:26:27.017550 9923 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 22:26:27.018409 11728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:27.007622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.252:37405 (local address 127.0.0.1:43604)
0808 22:26:27.007776 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0808 22:26:27.007789 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 22:26:27.007855 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:26:27.008630 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:26:27.008633 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:26:27.008792 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:26:27.008882 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:26:27.008943 (+ 61us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:26:27.010048 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:26:27.010052 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:26:27.010859 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:26:27.010869 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:26:27.011484 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:26:27.011488 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:26:27.011596 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:26:27.012019 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:26:27.012033 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:26:27.014343 (+ 2310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:26:27.016922 (+ 2579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:26:27.016926 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:26:27.016927 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:26:27.017158 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:26:27.017454 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:26:27.017457 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:26:27.017459 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:26:27.017554 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:26:27.017970 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:26:27.017973 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:26:27.018067 (+ 94us) client_negotiation.cc:769] Sending connection context
0808 22:26:27.018147 (+ 80us) client_negotiation.cc:240] Negotiation successful
0808 22:26:27.018271 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94}
I20260808 22:26:27.018412 11840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:27.007673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.176.252:37405 (local address 127.0.0.1:43588)
0808 22:26:27.007861 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0808 22:26:27.007873 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 22:26:27.007940 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:26:27.009512 (+ 1572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:26:27.009515 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:26:27.009701 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:26:27.009780 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:26:27.009786 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:26:27.010876 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:26:27.010878 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:26:27.011613 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:26:27.011622 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:26:27.012245 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:26:27.012249 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:26:27.012324 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:26:27.012848 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:26:27.012863 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:26:27.014468 (+ 1605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:26:27.017060 (+ 2592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:26:27.017063 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:26:27.017065 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:26:27.017288 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:26:27.017586 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:26:27.017588 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:26:27.017590 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:26:27.017668 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:26:27.018004 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:26:27.018007 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:26:27.018091 (+ 84us) client_negotiation.cc:769] Sending connection context
0808 22:26:27.018160 (+ 69us) client_negotiation.cc:240] Negotiation successful
0808 22:26:27.018274 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123}
I20260808 22:26:27.018927 11845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:27.008045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43604 (local address 127.9.176.252:37405)
0808 22:26:27.008360 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0808 22:26:27.008369 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 22:26:27.008383 (+ 14us) server_negotiation.cc:407] Connection header received
0808 22:26:27.008415 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:26:27.008417 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:26:27.008457 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:26:27.008602 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:26:27.009053 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:27.009918 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:27.011027 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:27.011366 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:27.014466 (+ 3100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:26:27.014573 (+ 107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:26:27.014576 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:26:27.014579 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:26:27.014598 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:26:27.016767 (+ 2169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:26:27.017271 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:26:27.017274 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:26:27.017276 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:26:27.017358 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:26:27.017659 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:26:27.017662 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:26:27.017664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:26:27.017853 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:26:27.018558 (+ 705us) server_negotiation.cc:1035] Waiting for connection context
0808 22:26:27.018644 (+ 86us) server_negotiation.cc:299] Negotiation successful
0808 22:26:27.018749 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"spinlock_wait_cycles":1792,"thread_start_us":102,"threads_started":1}
I20260808 22:26:27.018987 11843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:27.006999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43588 (local address 127.9.176.252:37405)
0808 22:26:27.009313 (+ 2314us) server_negotiation.cc:206] Beginning negotiation
0808 22:26:27.009324 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 22:26:27.009336 (+ 12us) server_negotiation.cc:407] Connection header received
0808 22:26:27.009366 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:26:27.009369 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:26:27.009406 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:26:27.009865 (+ 459us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:26:27.009964 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:27.010772 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:27.011769 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:27.012133 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:27.013416 (+ 1283us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:26:27.014573 (+ 1157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:26:27.014576 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:26:27.014579 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:26:27.014609 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:26:27.016801 (+ 2192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:26:27.017391 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:26:27.017393 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:26:27.017395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:26:27.017497 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:26:27.017749 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:26:27.017751 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:26:27.017753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:26:27.017920 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:26:27.018641 (+ 721us) server_negotiation.cc:1035] Waiting for connection context
0808 22:26:27.018713 (+ 72us) server_negotiation.cc:299] Negotiation successful
0808 22:26:27.018827 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":12,"server-negotiator.queue_time_us":421,"spinlock_wait_cycles":8576,"thread_start_us":95,"threads_started":1}
I20260808 22:26:27.019670 11617 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bdcf35097fa34f358c646a95b3f82c99: Neither blocks nor log segments found. Creating new log.
I20260808 22:26:27.018860 11586 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "bdcf35097fa34f358c646a95b3f82c99"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39354
I20260808 22:26:27.020542 11617 log.cc:824] T 00000000000000000000000000000000 P bdcf35097fa34f358c646a95b3f82c99: Log is configured to *not* fsync() on all Append() calls
I20260808 22:26:27.021823 11833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c095aa9f0f54125ad918ba8ebaa6be4: Bootstrap starting.
I20260808 22:26:27.022679 11617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bdcf35097fa34f358c646a95b3f82c99: No bootstrap required, opened a new log
I20260808 22:26:27.023206 11721 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3a4e460127749249a2d1e8098507182: Bootstrap starting.
I20260808 22:26:27.024287 11833 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c095aa9f0f54125ad918ba8ebaa6be4: Neither blocks nor log segments found. Creating new log.
I20260808 22:26:27.025179 11833 log.cc:824] T 00000000000000000000000000000000 P 5c095aa9f0f54125ad918ba8ebaa6be4: Log is configured to *not* fsync() on all Append() calls
I20260808 22:26:27.025627 11721 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3a4e460127749249a2d1e8098507182: Neither blocks nor log segments found. Creating new log.
I20260808 22:26:27.025663 11617 raft_consensus.cc:348] T 00000000000000000000000000000000 P bdcf35097fa34f358c646a95b3f82c99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcf35097fa34f358c646a95b3f82c99" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 40175 } } peers { permanent_uuid: "b3a4e460127749249a2d1e8098507182" member_type: VOTER last_known_addr { host: "127.9.176.253" port: 37351 } } peers { permanent_uuid: "5c095aa9f0f54125ad918ba8ebaa6be4" member_type: VOTER last_known_addr { host: "127.9.176.252" port: 37405 } }
I20260808 22:26:27.025843 11617 raft_consensus.cc:729] T 00000000000000000000000000000000 P bdcf35097fa34f358c646a95b3f82c99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdcf35097fa34f358c646a95b3f82c99, State: Initialized, Role: FOLLOWER
I20260808 22:26:27.026121 11617 consensus_queue.cc:260] T 00000000000000000000000000000000 P bdcf35097fa34f358c646a95b3f82c99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcf35097fa34f358c646a95b3f82c99" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 40175 } } peers { permanent_uuid: "b3a4e460127749249a2d1e8098507182" member_type: VOTER last_known_addr { host: "127.9.176.253" port: 37351 } } peers { permanent_uuid: "5c095aa9f0f54125ad918ba8ebaa6be4" member_type: VOTER last_known_addr { host: "127.9.176.252" port: 37405 } }
I20260808 22:26:27.026475 11721 log.cc:824] T 00000000000000000000000000000000 P b3a4e460127749249a2d1e8098507182: Log is configured to *not* fsync() on all Append() calls
I20260808 22:26:27.026587 11851 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdcf35097fa34f358c646a95b3f82c99 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcf35097fa34f358c646a95b3f82c99" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 40175 } } peers { permanent_uuid: "b3a4e460127749249a2d1e8098507182" member_type: VOTER last_known_addr { host: "127.9.176.253" port: 37351 } } peers { permanent_uuid: "5c095aa9f0f54125ad918ba8ebaa6be4" member_type: VOTER last_known_addr { host: "127.9.176.252" port: 37405 } } }
I20260808 22:26:27.026722 11851 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdcf35097fa34f358c646a95b3f82c99 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 22:26:27.027410 11617 sys_catalog.cc:564] T 00000000000000000000000000000000 P bdcf35097fa34f358c646a95b3f82c99 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:26:27.027890 11833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c095aa9f0f54125ad918ba8ebaa6be4: No bootstrap required, opened a new log
I20260808 22:26:27.028576 11721 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3a4e460127749249a2d1e8098507182: No bootstrap required, opened a new log
W20260808 22:26:27.032361 11876 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bdcf35097fa34f358c646a95b3f82c99: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 22:26:27.032415 11876 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 22:26:27.032501 11833 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c095aa9f0f54125ad918ba8ebaa6be4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcf35097fa34f358c646a95b3f82c99" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 40175 } } peers { permanent_uuid: "b3a4e460127749249a2d1e8098507182" member_type: VOTER last_known_addr { host: "127.9.176.253" port: 37351 } } peers { permanent_uuid: "5c095aa9f0f54125ad918ba8ebaa6be4" member_type: VOTER last_known_addr { host: "127.9.176.252" port: 37405 } }
I20260808 22:26:27.032730 11833 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c095aa9f0f54125ad918ba8ebaa6be4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c095aa9f0f54125ad918ba8ebaa6be4, State: Initialized, Role: FOLLOWER
I20260808 22:26:27.032985 11586 raft_consensus.cc:482] T 00000000000000000000000000000000 P bdcf35097fa34f358c646a95b3f82c99 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 22:26:27.033100 11586 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bdcf35097fa34f358c646a95b3f82c99 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:26:27.032980 11721 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3a4e460127749249a2d1e8098507182 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcf35097fa34f358c646a95b3f82c99" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 40175 } } peers { permanent_uuid: "b3a4e460127749249a2d1e8098507182" member_type: VOTER last_known_addr { host: "127.9.176.253" port: 37351 } } peers { permanent_uuid: "5c095aa9f0f54125ad918ba8ebaa6be4" member_type: VOTER last_known_addr { host: "127.9.176.252" port: 37405 } }
I20260808 22:26:27.033190 11721 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3a4e460127749249a2d1e8098507182 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3a4e460127749249a2d1e8098507182, State: Initialized, Role: FOLLOWER
I20260808 22:26:27.033483 11721 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3a4e460127749249a2d1e8098507182 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcf35097fa34f358c646a95b3f82c99" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 40175 } } peers { permanent_uuid: "b3a4e460127749249a2d1e8098507182" member_type: VOTER last_known_addr { host: "127.9.176.253" port: 37351 } } peers { permanent_uuid: "5c095aa9f0f54125ad918ba8ebaa6be4" member_type: VOTER last_known_addr { host: "127.9.176.252" port: 37405 } }
I20260808 22:26:27.034365 11586 raft_consensus.cc:504] T 00000000000000000000000000000000 P bdcf35097fa34f358c646a95b3f82c99 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcf35097fa34f358c646a95b3f82c99" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 40175 } } peers { permanent_uuid: "b3a4e460127749249a2d1e8098507182" member_type: VOTER last_known_addr { host: "127.9.176.253" port: 37351 } } peers { permanent_uuid: "5c095aa9f0f54125ad918ba8ebaa6be4" member_type: VOTER last_known_addr { host: "127.9.176.252" port: 37405 } }
I20260808 22:26:27.034741 11721 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3a4e460127749249a2d1e8098507182 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:26:27.032146 11520 master_runner.cc:186] Error getting master registration for 127.9.176.253:37351: OK, instance_id { permanent_uuid: "b3a4e460127749249a2d1e8098507182" instance_seqno: 1786227986752106 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 22:26:27.035108 11520 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 22:26:27.035264 11586 leader_election.cc:288] T 00000000000000000000000000000000 P bdcf35097fa34f358c646a95b3f82c99 [CANDIDATE]: Term 1 election: Requested vote from peers b3a4e460127749249a2d1e8098507182 (127.9.176.253:37351), 5c095aa9f0f54125ad918ba8ebaa6be4 (127.9.176.252:37405)
I20260808 22:26:27.036110 11685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bdcf35097fa34f358c646a95b3f82c99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b3a4e460127749249a2d1e8098507182"
I20260808 22:26:27.036137 11833 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c095aa9f0f54125ad918ba8ebaa6be4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcf35097fa34f358c646a95b3f82c99" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 40175 } } peers { permanent_uuid: "b3a4e460127749249a2d1e8098507182" member_type: VOTER last_known_addr { host: "127.9.176.253" port: 37351 } } peers { permanent_uuid: "5c095aa9f0f54125ad918ba8ebaa6be4" member_type: VOTER last_known_addr { host: "127.9.176.252" port: 37405 } }
I20260808 22:26:27.036340 11685 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3a4e460127749249a2d1e8098507182 [term 0 FOLLOWER]: Advancing to term 1
W20260808 22:26:27.036863 11551 tablet.cc:2369] T 00000000000000000000000000000000 P bdcf35097fa34f358c646a95b3f82c99: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 22:26:27.037599 11897 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c095aa9f0f54125ad918ba8ebaa6be4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcf35097fa34f358c646a95b3f82c99" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 40175 } } peers { permanent_uuid: "b3a4e460127749249a2d1e8098507182" member_type: VOTER last_known_addr { host: "127.9.176.253" port: 37351 } } peers { permanent_uuid: "5c095aa9f0f54125ad918ba8ebaa6be4" member_type: VOTER last_known_addr { host: "127.9.176.252" port: 37405 } } }
I20260808 22:26:27.037703 11685 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b3a4e460127749249a2d1e8098507182 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdcf35097fa34f358c646a95b3f82c99 in term 1.
I20260808 22:26:27.037734 11897 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c095aa9f0f54125ad918ba8ebaa6be4 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 22:26:27.038137 11546 leader_election.cc:302] T 00000000000000000000000000000000 P bdcf35097fa34f358c646a95b3f82c99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3a4e460127749249a2d1e8098507182, bdcf35097fa34f358c646a95b3f82c99; no voters:
I20260808 22:26:27.038435 11833 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c095aa9f0f54125ad918ba8ebaa6be4 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:26:27.038465 11799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bdcf35097fa34f358c646a95b3f82c99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5c095aa9f0f54125ad918ba8ebaa6be4"
I20260808 22:26:27.038688 11799 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c095aa9f0f54125ad918ba8ebaa6be4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:26:27.039953 11851 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bdcf35097fa34f358c646a95b3f82c99 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:26:27.039952 11799 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5c095aa9f0f54125ad918ba8ebaa6be4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdcf35097fa34f358c646a95b3f82c99 in term 1.
I20260808 22:26:27.040539 11851 raft_consensus.cc:686] T 00000000000000000000000000000000 P bdcf35097fa34f358c646a95b3f82c99 [term 1 LEADER]: Becoming Leader. State: Replica: bdcf35097fa34f358c646a95b3f82c99, State: Running, Role: LEADER
I20260808 22:26:27.040864 11851 consensus_queue.cc:237] T 00000000000000000000000000000000 P bdcf35097fa34f358c646a95b3f82c99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcf35097fa34f358c646a95b3f82c99" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 40175 } } peers { permanent_uuid: "b3a4e460127749249a2d1e8098507182" member_type: VOTER last_known_addr { host: "127.9.176.253" port: 37351 } } peers { permanent_uuid: "5c095aa9f0f54125ad918ba8ebaa6be4" member_type: VOTER last_known_addr { host: "127.9.176.252" port: 37405 } }
I20260808 22:26:27.041039 11878 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3a4e460127749249a2d1e8098507182 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcf35097fa34f358c646a95b3f82c99" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 40175 } } peers { permanent_uuid: "b3a4e460127749249a2d1e8098507182" member_type: VOTER last_known_addr { host: "127.9.176.253" port: 37351 } } peers { permanent_uuid: "5c095aa9f0f54125ad918ba8ebaa6be4" member_type: VOTER last_known_addr { host: "127.9.176.252" port: 37405 } } }
I20260808 22:26:27.041174 11878 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3a4e460127749249a2d1e8098507182 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 22:26:27.042429 11851 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdcf35097fa34f358c646a95b3f82c99 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bdcf35097fa34f358c646a95b3f82c99. Latest consensus state: current_term: 1 leader_uuid: "bdcf35097fa34f358c646a95b3f82c99" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcf35097fa34f358c646a95b3f82c99" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 40175 } } peers { permanent_uuid: "b3a4e460127749249a2d1e8098507182" member_type: VOTER last_known_addr { host: "127.9.176.253" port: 37351 } } peers { permanent_uuid: "5c095aa9f0f54125ad918ba8ebaa6be4" member_type: VOTER last_known_addr { host: "127.9.176.252" port: 37405 } } }
I20260808 22:26:27.042582 11851 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdcf35097fa34f358c646a95b3f82c99 [sys.catalog]: This master's current role is: LEADER
I20260808 22:26:27.042924 11925 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:26:27.045169 11925 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260808 22:26:27.049037 11923 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5c095aa9f0f54125ad918ba8ebaa6be4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 22:26:27.049114 11923 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260808 22:26:27.051142 11929 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b3a4e460127749249a2d1e8098507182: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 22:26:27.051226 11929 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 22:26:27.053503 11799 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5c095aa9f0f54125ad918ba8ebaa6be4 [term 1 FOLLOWER]: Refusing update from remote peer bdcf35097fa34f358c646a95b3f82c99: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:26:27.054188 11922 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bdcf35097fa34f358c646a95b3f82c99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c095aa9f0f54125ad918ba8ebaa6be4" member_type: VOTER last_known_addr { host: "127.9.176.252" port: 37405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:26:27.057924 11897 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c095aa9f0f54125ad918ba8ebaa6be4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bdcf35097fa34f358c646a95b3f82c99. Latest consensus state: current_term: 1 leader_uuid: "bdcf35097fa34f358c646a95b3f82c99" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcf35097fa34f358c646a95b3f82c99" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 40175 } } peers { permanent_uuid: "b3a4e460127749249a2d1e8098507182" member_type: VOTER last_known_addr { host: "127.9.176.253" port: 37351 } } peers { permanent_uuid: "5c095aa9f0f54125ad918ba8ebaa6be4" member_type: VOTER last_known_addr { host: "127.9.176.252" port: 37405 } } }
I20260808 22:26:27.058045 11897 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c095aa9f0f54125ad918ba8ebaa6be4 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 22:26:27.058974 11685 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b3a4e460127749249a2d1e8098507182 [term 1 FOLLOWER]: Refusing update from remote peer bdcf35097fa34f358c646a95b3f82c99: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:26:27.059540 11922 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bdcf35097fa34f358c646a95b3f82c99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3a4e460127749249a2d1e8098507182" member_type: VOTER last_known_addr { host: "127.9.176.253" port: 37351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:26:27.062902 11878 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3a4e460127749249a2d1e8098507182 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bdcf35097fa34f358c646a95b3f82c99. Latest consensus state: current_term: 1 leader_uuid: "bdcf35097fa34f358c646a95b3f82c99" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcf35097fa34f358c646a95b3f82c99" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 40175 } } peers { permanent_uuid: "b3a4e460127749249a2d1e8098507182" member_type: VOTER last_known_addr { host: "127.9.176.253" port: 37351 } } peers { permanent_uuid: "5c095aa9f0f54125ad918ba8ebaa6be4" member_type: VOTER last_known_addr { host: "127.9.176.252" port: 37405 } } }
I20260808 22:26:27.063024 11878 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3a4e460127749249a2d1e8098507182 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 22:26:27.065531 11922 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdcf35097fa34f358c646a95b3f82c99 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bdcf35097fa34f358c646a95b3f82c99" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcf35097fa34f358c646a95b3f82c99" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 40175 } } peers { permanent_uuid: "b3a4e460127749249a2d1e8098507182" member_type: VOTER last_known_addr { host: "127.9.176.253" port: 37351 } } peers { permanent_uuid: "5c095aa9f0f54125ad918ba8ebaa6be4" member_type: VOTER last_known_addr { host: "127.9.176.252" port: 37405 } } }
I20260808 22:26:27.065668 11922 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdcf35097fa34f358c646a95b3f82c99 [sys.catalog]: This master's current role is: LEADER
I20260808 22:26:27.066567 11878 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3a4e460127749249a2d1e8098507182 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bdcf35097fa34f358c646a95b3f82c99" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcf35097fa34f358c646a95b3f82c99" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 40175 } } peers { permanent_uuid: "b3a4e460127749249a2d1e8098507182" member_type: VOTER last_known_addr { host: "127.9.176.253" port: 37351 } } peers { permanent_uuid: "5c095aa9f0f54125ad918ba8ebaa6be4" member_type: VOTER last_known_addr { host: "127.9.176.252" port: 37405 } } }
I20260808 22:26:27.066666 11878 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3a4e460127749249a2d1e8098507182 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 22:26:27.067058 11935 mvcc.cc:205] Tried to move back new op lower bound from 7316389834965159936 to 7316389834922627072. Current Snapshot: MvccSnapshot[applied={T|T < 7316389834965159936}]
I20260808 22:26:27.070338 11842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:27.044191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39366 (local address 127.9.176.254:40175)
0808 22:26:27.044304 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0808 22:26:27.044309 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:26:27.051525 (+ 7216us) server_negotiation.cc:407] Connection header received
0808 22:26:27.054058 (+ 2533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:26:27.054061 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:26:27.054112 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:26:27.054163 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:26:27.055866 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:27.056671 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:27.062916 (+ 6245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:27.063283 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:27.063361 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:26:27.066603 (+ 3242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:26:27.066607 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:26:27.066610 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:26:27.066636 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:26:27.068491 (+ 1855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:26:27.069013 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:26:27.069017 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:26:27.069019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:26:27.069105 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:26:27.069395 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:26:27.069398 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:26:27.069400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:26:27.069607 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:26:27.069712 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0808 22:26:27.070067 (+ 355us) server_negotiation.cc:299] Negotiation successful
0808 22:26:27.070187 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260808 22:26:27.073331 11843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:27.045414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43616 (local address 127.9.176.252:37405)
0808 22:26:27.046112 (+ 698us) server_negotiation.cc:206] Beginning negotiation
0808 22:26:27.046117 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:26:27.051927 (+ 5810us) server_negotiation.cc:407] Connection header received
0808 22:26:27.053808 (+ 1881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:26:27.053812 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:26:27.053860 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:26:27.053910 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:26:27.057902 (+ 3992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:27.058722 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:27.061742 (+ 3020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:27.062118 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:27.062215 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:26:27.069341 (+ 7126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:26:27.069345 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:26:27.069348 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:26:27.069376 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:26:27.071077 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:26:27.071922 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:26:27.071925 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:26:27.071927 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:26:27.072016 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:26:27.072327 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:26:27.072331 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:26:27.072333 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:26:27.072543 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:26:27.072669 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0808 22:26:27.073054 (+ 385us) server_negotiation.cc:299] Negotiation successful
0808 22:26:27.073178 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619}
I20260808 22:26:27.075155 11730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:27.044961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40720 (local address 127.9.176.253:37351)
0808 22:26:27.045078 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0808 22:26:27.045083 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:26:27.051739 (+ 6656us) server_negotiation.cc:407] Connection header received
0808 22:26:27.052286 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:26:27.052290 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:26:27.052337 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:26:27.052384 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:26:27.059859 (+ 7475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:27.060645 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:27.063985 (+ 3340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:27.064345 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:27.064423 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:26:27.071581 (+ 7158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:26:27.071585 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:26:27.071588 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:26:27.071615 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:26:27.073240 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:26:27.073896 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:26:27.073900 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:26:27.073902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:26:27.073996 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:26:27.074267 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:26:27.074271 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:26:27.074272 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:26:27.074468 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:26:27.074568 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0808 22:26:27.074879 (+ 311us) server_negotiation.cc:299] Negotiation successful
0808 22:26:27.075010 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260808 22:26:27.078756 11851 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdcf35097fa34f358c646a95b3f82c99 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bdcf35097fa34f358c646a95b3f82c99" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcf35097fa34f358c646a95b3f82c99" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 40175 } } peers { permanent_uuid: "b3a4e460127749249a2d1e8098507182" member_type: VOTER last_known_addr { host: "127.9.176.253" port: 37351 } } peers { permanent_uuid: "5c095aa9f0f54125ad918ba8ebaa6be4" member_type: VOTER last_known_addr { host: "127.9.176.252" port: 37405 } } }
I20260808 22:26:27.078886 11851 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdcf35097fa34f358c646a95b3f82c99 [sys.catalog]: This master's current role is: LEADER
I20260808 22:26:27.080075 11897 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c095aa9f0f54125ad918ba8ebaa6be4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bdcf35097fa34f358c646a95b3f82c99" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcf35097fa34f358c646a95b3f82c99" member_type: VOTER last_known_addr { host: "127.9.176.254" port: 40175 } } peers { permanent_uuid: "b3a4e460127749249a2d1e8098507182" member_type: VOTER last_known_addr { host: "127.9.176.253" port: 37351 } } peers { permanent_uuid: "5c095aa9f0f54125ad918ba8ebaa6be4" member_type: VOTER last_known_addr { host: "127.9.176.252" port: 37405 } } }
I20260808 22:26:27.080184 11897 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c095aa9f0f54125ad918ba8ebaa6be4 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 22:26:27.081498 11940 mvcc.cc:205] Tried to move back new op lower bound from 7316389834965159936 to 7316389834922627072. Current Snapshot: MvccSnapshot[applied={T|T < 7316389834965159936}]
I20260808 22:26:27.083318 11925 catalog_manager.cc:1347] Generated new cluster ID: 525845c2663048ca8204f824966159ac
I20260808 22:26:27.083386 11925 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:26:27.111812 11925 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:26:27.113230 11925 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:26:27.137061 11925 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bdcf35097fa34f358c646a95b3f82c99: Generated new TSK 0
I20260808 22:26:27.137974 11925 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 22:26:27.166256 11842 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:26:27.163168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39382 (local address 127.9.176.254:40175)
0808 22:26:27.163313 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0808 22:26:27.163319 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 22:26:27.165146 (+ 1827us) server_negotiation.cc:407] Connection header received
0808 22:26:27.165276 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:26:27.165280 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:26:27.165337 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:26:27.165430 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:26:27.166121 (+ 691us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39382: BlockingRecv error: recv got EOF from 127.0.0.1:39382 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260808 22:26:27.166405 11730 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:26:27.163857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40732 (local address 127.9.176.253:37351)
0808 22:26:27.163989 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0808 22:26:27.163994 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:26:27.165430 (+ 1436us) server_negotiation.cc:407] Connection header received
0808 22:26:27.165593 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:26:27.165596 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:26:27.165645 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:26:27.165694 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:26:27.166275 (+ 581us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40732: BlockingRecv error: recv got EOF from 127.0.0.1:40732 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260808 22:26:27.167883 11843 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:26:27.164495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43622 (local address 127.9.176.252:37405)
0808 22:26:27.164609 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0808 22:26:27.164614 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:26:27.166893 (+ 2279us) server_negotiation.cc:407] Connection header received
0808 22:26:27.167132 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:26:27.167136 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:26:27.167187 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:26:27.167236 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:26:27.167764 (+ 528us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43622: BlockingRecv error: recv got EOF from 127.0.0.1:43622 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260808 22:26:27.171612 11730 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:26:27.170144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40748 (local address 127.9.176.253:37351)
0808 22:26:27.170240 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0808 22:26:27.170244 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:26:27.170689 (+ 445us) server_negotiation.cc:407] Connection header received
0808 22:26:27.170902 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:26:27.170906 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:26:27.170981 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:26:27.171043 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:26:27.171514 (+ 471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40748: BlockingRecv error: recv got EOF from 127.0.0.1:40748 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260808 22:26:27.176683 11730 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:26:27.175220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40752 (local address 127.9.176.253:37351)
0808 22:26:27.175315 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0808 22:26:27.175319 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:26:27.175814 (+ 495us) server_negotiation.cc:407] Connection header received
0808 22:26:27.175994 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:26:27.175997 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:26:27.176059 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:26:27.176105 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:26:27.176579 (+ 474us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40752: BlockingRecv error: recv got EOF from 127.0.0.1:40752 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260808 22:26:27.182269 11730 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:26:27.180772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40762 (local address 127.9.176.253:37351)
0808 22:26:27.180861 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0808 22:26:27.180865 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:26:27.181342 (+ 477us) server_negotiation.cc:407] Connection header received
0808 22:26:27.181501 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:26:27.181504 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:26:27.181550 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:26:27.181596 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:26:27.182047 (+ 451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40762: BlockingRecv error: recv got EOF from 127.0.0.1:40762 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260808 22:26:27.189859 11730 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:26:27.188326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40764 (local address 127.9.176.253:37351)
0808 22:26:27.188457 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0808 22:26:27.188461 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:26:27.188960 (+ 499us) server_negotiation.cc:407] Connection header received
0808 22:26:27.189162 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:26:27.189166 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:26:27.189215 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:26:27.189261 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:26:27.189758 (+ 497us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40764: BlockingRecv error: recv got EOF from 127.0.0.1:40764 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260808 22:26:27.202039 11730 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:26:27.200311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40768 (local address 127.9.176.253:37351)
0808 22:26:27.200427 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0808 22:26:27.200432 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:26:27.201033 (+ 601us) server_negotiation.cc:407] Connection header received
0808 22:26:27.201251 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:26:27.201254 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:26:27.201314 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:26:27.201368 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:26:27.201912 (+ 544us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40768: BlockingRecv error: recv got EOF from 127.0.0.1:40768 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260808 22:26:27.223057 11730 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:26:27.220846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40772 (local address 127.9.176.253:37351)
0808 22:26:27.221012 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0808 22:26:27.221017 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:26:27.221828 (+ 811us) server_negotiation.cc:407] Connection header received
0808 22:26:27.222088 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:26:27.222092 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:26:27.222156 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:26:27.222239 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:26:27.222919 (+ 680us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40772: BlockingRecv error: recv got EOF from 127.0.0.1:40772 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260808 22:26:27.258590 11730 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:26:27.257588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40780 (local address 127.9.176.253:37351)
0808 22:26:27.257724 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0808 22:26:27.257729 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:26:27.257803 (+ 74us) server_negotiation.cc:407] Connection header received
0808 22:26:27.257929 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:26:27.257932 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:26:27.257989 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:26:27.258080 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:26:27.258463 (+ 383us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40780: BlockingRecv error: recv got EOF from 127.0.0.1:40780 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260808 22:26:27.325851 11730 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:26:27.324655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40784 (local address 127.9.176.253:37351)
0808 22:26:27.324812 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0808 22:26:27.324817 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:26:27.324944 (+ 127us) server_negotiation.cc:407] Connection header received
0808 22:26:27.325134 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:26:27.325137 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:26:27.325186 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:26:27.325255 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:26:27.325743 (+ 488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40784: BlockingRecv error: recv got EOF from 127.0.0.1:40784 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260808 22:26:27.457039 11730 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:26:27.455863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40796 (local address 127.9.176.253:37351)
0808 22:26:27.456035 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0808 22:26:27.456040 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:26:27.456133 (+ 93us) server_negotiation.cc:407] Connection header received
0808 22:26:27.456292 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:26:27.456295 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:26:27.456350 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:26:27.456422 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:26:27.456919 (+ 497us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40796: BlockingRecv error: recv got EOF from 127.0.0.1:40796 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260808 22:26:27.716148 11730 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:26:27.715117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40798 (local address 127.9.176.253:37351)
0808 22:26:27.715242 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0808 22:26:27.715246 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:26:27.715275 (+ 29us) server_negotiation.cc:407] Connection header received
0808 22:26:27.715414 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:26:27.715416 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:26:27.715463 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:26:27.715561 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:26:27.716037 (+ 476us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40798: BlockingRecv error: recv got EOF from 127.0.0.1:40798 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 22:26:28.050401 11923 catalog_manager.cc:1259] Loaded cluster ID: 525845c2663048ca8204f824966159ac
I20260808 22:26:28.050488 11923 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5c095aa9f0f54125ad918ba8ebaa6be4: loading cluster ID for follower catalog manager: success
I20260808 22:26:28.052485 11929 catalog_manager.cc:1259] Loaded cluster ID: 525845c2663048ca8204f824966159ac
I20260808 22:26:28.052516 11923 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5c095aa9f0f54125ad918ba8ebaa6be4: acquiring CA information for follower catalog manager: success
I20260808 22:26:28.052559 11929 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b3a4e460127749249a2d1e8098507182: loading cluster ID for follower catalog manager: success
I20260808 22:26:28.053956 11923 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5c095aa9f0f54125ad918ba8ebaa6be4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 22:26:28.054450 11929 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b3a4e460127749249a2d1e8098507182: acquiring CA information for follower catalog manager: success
I20260808 22:26:28.055409 11929 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b3a4e460127749249a2d1e8098507182: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 22:26:28.233876 12045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:28.230135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40808 (local address 127.9.176.253:37351)
0808 22:26:28.230408 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0808 22:26:28.230411 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:26:28.230428 (+ 17us) server_negotiation.cc:407] Connection header received
0808 22:26:28.230598 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:26:28.230600 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:26:28.230727 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:26:28.230825 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:26:28.231078 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:28.231607 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:28.232458 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:28.232712 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:28.232792 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:26:28.233104 (+ 312us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:26:28.233556 (+ 452us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:26:28.233653 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0808 22:26:28.233681 (+ 28us) server_negotiation.cc:299] Negotiation successful
0808 22:26:28.233727 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":87,"threads_started":1}
I20260808 22:26:28.240316 12051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:28.236776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43638 (local address 127.9.176.252:37405)
0808 22:26:28.237034 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0808 22:26:28.237039 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:26:28.237053 (+ 14us) server_negotiation.cc:407] Connection header received
0808 22:26:28.237144 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:26:28.237147 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:26:28.237257 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:26:28.237355 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:26:28.237565 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:28.238085 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:28.238943 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:28.239182 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:28.239268 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:26:28.239589 (+ 321us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:26:28.239998 (+ 409us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:26:28.240093 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0808 22:26:28.240155 (+ 62us) server_negotiation.cc:299] Negotiation successful
0808 22:26:28.240202 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":62,"threads_started":1}
I20260808 22:26:28.247030 12045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:28.243297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40822 (local address 127.9.176.253:37351)
0808 22:26:28.243416 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0808 22:26:28.243420 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:26:28.243710 (+ 290us) server_negotiation.cc:407] Connection header received
0808 22:26:28.243812 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:26:28.243815 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:26:28.243861 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:26:28.243919 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:26:28.244133 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:28.244877 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:28.246061 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:28.246294 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:28.246385 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:26:28.246614 (+ 229us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:26:28.246683 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:26:28.246738 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0808 22:26:28.246838 (+ 100us) server_negotiation.cc:299] Negotiation successful
0808 22:26:28.246886 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260808 22:26:28.247394 12051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:28.243650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43646 (local address 127.9.176.252:37405)
0808 22:26:28.243807 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0808 22:26:28.243811 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:26:28.243822 (+ 11us) server_negotiation.cc:407] Connection header received
0808 22:26:28.243863 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:26:28.243865 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:26:28.243906 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:26:28.243965 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:26:28.244279 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:28.245033 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:28.246219 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:28.246518 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:28.246583 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:26:28.246929 (+ 346us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:26:28.247022 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:26:28.247070 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0808 22:26:28.247232 (+ 162us) server_negotiation.cc:299] Negotiation successful
0808 22:26:28.247283 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260808 22:26:28.248037 12062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:26:28.243144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39396 (local address 127.9.176.254:40175)
0808 22:26:28.243982 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0808 22:26:28.243987 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:26:28.243999 (+ 12us) server_negotiation.cc:407] Connection header received
0808 22:26:28.244101 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:26:28.244104 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:26:28.244261 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:26:28.244367 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:26:28.244579 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:28.245265 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:28.246417 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:26:28.246791 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:26:28.246886 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:26:28.247279 (+ 393us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:26:28.247609 (+ 330us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:26:28.247695 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0808 22:26:28.247828 (+ 133us) server_negotiation.cc:299] Negotiation successful
0808 22:26:28.247885 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":763,"thread_start_us":67,"threads_started":1}
I20260808 22:26:28.250877 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 11520
I20260808 22:26:28.264868 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 11627
I20260808 22:26:28.274667 9923 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqqDyFd/build/debug/bin/kudu with pid 11733
2026-08-08T22:26:28Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3163 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6322 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (281966 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST